argh

I was playing around with some python code a couple months ago. Nothing that special, just a few hours work figuring out how to parse some text files. Nothing too elaborate, but I remember being particularly pleased with one or two pieces of it.

Finally getting back to it tonight and it's nowhere to be found... I've found tons of unrelated crap I saved off and will never use again. The time I'm not anal about putting some experiments into version control is also the time I deleted it without realizing it.

I think it must have also been when I was fiddling with eclipse+pydev+IronPython, and I hadn't bothered to change the project's path from the default place in the default Eclipse workspace. Normally I always change one or both of those...

Sometime in the last few weeks I must have done my usual iteration of "eclipse is acting a little wonky, time to delete it and re-install it and all my usual plugins. Oh, and this workspace directory I never use."

Argh.

I also distinctly remember thinking that I should make a blog post of some of the code. Didn't happen.

Double argh.

fall 2009 status

• dog still good. Other than chomping on the neighbor's cat.
• new job still good. Fixed a long-outstanding bug in the code. Waiting to get more bugs to work on, and in the meantime working on a new tool to help ease our transition from ClearCase to Subversion. Getting to play in python most of the day, good times.
• Best book I've read lately: Soon I Will Be Invincible, by Austin Grossman.
• Best audiobook I've listened to lately: How to Succeed in Evil, by Patrick McLean

Odd that both the book and the audiobook are superhero satire/homage.

Currently reading: Matter, by Iain M. Banks

losing my geek cred

Finding pydev + eclipse a decent IDE for playing around with python. That place in my heart used to be held by Vim.

The very new IronPython integration in pydev is surprisingly functional.

interesting dynamic languages on .net talk from 2008

why I love python

Lately, discussions at the NFJS conference and recent conversations with friends have made me introspective about why I like Python.

I first started using Python about 9 years ago. Previously, I'd used mostly C, C++, Perl, Pascal, and a very little bit of Visual Basic.

Pascal was what I'd learned in High School. VB, C, and C++ were what I'd used during college (I guess there was a teeeeny bit of Lisp thrown in there too). My first couple years as a software engineer, C++ was the hammer with which I'd pounded all programming nails.

Eventually I learned Perl. It was OK. But, like most people, I'd write Perl scripts and not be able to read them the next month.

After a recomendation I'd read online, I tried the Python tutorial. The conciseness was a revelation. The way it got out of my way and let me worry about the problem and not the compile/run/debug loop. Having dictionaries and lists as a first-class language features blew me away. I don't know why similar features in Perl never struck me in the same way.

Learning Python not only completely reinvigorated my interest in programming, but it also and completely changed the way I program. Maybe I still love Python because of the sentimental attachment to it from that first revelatory period of experimentation.

no fluff just stuff SLC 2009

It was pretty good. Only went to one session that was a dud. Decided to try to go to a number of different speakers in an attempt to not get burnt out. But, I should have just stuck with all of Venkat Subramanian's sessions the first day's afternoon. He's an awesome speaker.

I enjoyed seeing other dynamic languages -- groovy, scala, clojure. Definitely seems like mixing in Groovy to any java project would be pretty easy. Loved that it removed a lot of Java's verbosity, but kept the barrier of entry low. Groovy also seemed to be much more readable than clojure or scala.

Scala seemed very interesting -- liked the type inference, and concurrency support. But, I hate hate hate the underscore. And that g-damn colon that changed the operator/operand. Maybe if I used it every day, it'd seem more readable.

I guess if people can hate Python's significant whitespace (which I love), I can have an irrational hatred of Scala's underscores/colons.

python + elixir + pyyaml == yay!

And so... I'm playing with a little toy script to ease some pain at work. We've got a bunch of bills in a very large database, the bill data our application reads/modifies is split across a number of tables.

At the moment, the developers manually run SQL scripts to peek at various bits of data. Definitely a huge waste of time.

So, I'm playing with Python + Elixir + PyYAML to get a script that I can just pass in the bill ID and it'll query the 5-6 tables and serialize the bill to something more human readable. If I play my cards right, I'll never have to deal with Toad again.

Elixir works its magic and I can query my database. The autoload didn't work, and there are a couple tables with 50-100+ columns. Serializing it to YAML without repeating myself is now the trick.

I'm sure there's a better way to do this, but here's what I came up with... applied to the Elixir tutorial. Not the most exciting thing ever. But, it's the start of building a better testing framework. Doing the same for the real tables will make it much easier to compare a bill before and after running through our application.

#!/usr/bin/python

# -*- coding: latin-1 -*-



from elixir import *



from yaml import load, dump

try:

    from yaml import CLoader as Loader

    from yaml import CDumper as Dumper

except ImportError:

    from yaml import Loader, Dumper





def _toYamlRep(ent):

    """

    Given an elixir entity, query the entity's members via its __dict__ 

    and return a dict

    """

    ret = {}

    for (k,v) in ent.__dict__.items():

        if k.startswith('_') or k == 'row_type':

            # don't print out the 'hidden' keys

            continue

        if v:

            ret[k] = str(v)

    return ret



class YamlEntity(Entity):

    def toYamlRep(self):

        """

        Wrap the _toYamlRep() dict in another dict, use the class' name as the header.

        """

        return {self.__class__.__name__ : _toYamlRep(self) }



class Movie(YamlEntity):

    title = Field(String(30))

    year = Field(Integer())

    description = Field(Text())

    director = ManyToOne('Director')



    def __repr__(self):

        return '<Movie: "%s" (%d)>'%(self.title,self.year)





class Director(YamlEntity):

    name = Field(String(60))

    movies = OneToMany('Movie')



    def __repr__(self):

        return '<Director: "%s">'%(self.name)





def main():

    metadata.bind = "sqlite://"

    setup_all()

    create_all()



    rscott = Director(name="Ridley Scott")

    glucas = Director(name="George Lucas")

    alien = Movie(title="Alien", year=1979, director=rscott)

    swars = Movie(title="Star Wars", year=1977, director=glucas)

    brunner = Movie(title="Blade Runner", year=1982, director=rscott)

    

    session.flush()



    for m in Movie.query().all():

        print dump(m.toYamlRep(),Dumper=Dumper,default_flow_style=False)



    cleanup_all()



if __name__ == '__main__':

    main()

And, here's the output:

Movie:
  director: '<Director: "George Lucas">'
  director_id: '2'
  id: '3'
  title: Star Wars
  year: '1977'

Movie:
  director: '<Director: "Ridley Scott">'
  director_id: '1'
  id: '4'
  title: Alien
  year: '1979'

Movie:
  director: '<Director: "Ridley Scott">'
  director_id: '1'
  id: '5'
  title: Blade Runner
  year: '1982'

Salt Lake Software Symposium, day #2

Last day. Still pretty good, but didn't rock as much as the first.

Attended a couple more of Jared Richardson's sessions: "Agile Testing Strategies", and "Software Development Techniques". Lots of good real-world examples, and lots of inspiration for changing our processes. Definitely thinking of picking up Ship It!

Neal Ford's "Pragmatic Extreme Programming" was good. But, I think it'll be easier to work in lessons from Jared's sessions (on days 1 and 2) into our existing process.

I wasn't going to attend Brian Sam-Bodden's "Complex Builds with Ant" session. From the slides, it appeared like I'd already knew the major tips on my own. But, I was wiped out by the last session. Decided it was worth going to a topic I was familiar with in order to pick an expert's brain. Only three attendees, so the session went fast and I was able to ask a lot of questions.

More sour-grapes today over all the Ruby rah-rah-rah. Again, Neal Ford declared Ruby the winner in the dynamic language race on the JVM. Neal is a very smart guy, a great presenter, and also great to talk to one-on-one. I don't have any issue with him advocating Ruby. Any language that makes developers more productive is a good thing. But, it seems disingenuous to declare a victor in a wide open race to a room full of people who have little to no experience with dynamic languages.

Personally, I prefer Python over Ruby. Others can and should disagree. Here's a fair, but slightly biased (since the author is familiar with Python), comparison of Python vs. Ruby.

I'm not sure who to blame for lack of Python excitement in the Java world (or at least at this conference). Maybe with the recent revitalization of the Jython project, it will get more Java developer's attention.

Microsoft's CLR (and the new DLR and Silverlight) seems very exciting. See here and here and here. The second link is a screencast showing interoperation between Ruby, JavaScript, Python, and VB.

All in all, the conference was very good. I'll definitely plan on going next year.

Salt Lake Software Symposium, day #1

Since this blog is all about (pick one):

• lame book reviews
• lame software conference anecdotes
• odd things I've put in my mouth
  

Here's more software conference anecdotes.

Spent today at the Salt Lake Software Symposium (aka No Fluff Just Stuff). So far, the conference is very good. Attended presentations by Neal Ford, Jared Richardson, and Brian Sletten. All great presenters, showing off great technology.

A friend has tried 2-3 times to explain Aspect Oriented Programming to me. It sounded... like more trouble than it was worth. Brian Sletten's AOP presentation was very useful in cluing me in. Seeing the AspectJ/Eclipse tools in action got me all excited to try it out. The immediate use-case I have would to remove a bunch of cut-n-paste concurrent locking boilerplate in my pipeline application. If I understood it correctly (always a questionable assumption), I can replace all that code scattered throughout my class with an aspect that'll do the

  writeLock();
  try {
       // .... do stuff....
  } finally {
       releaseWriteLock();
  }

logic all from a single place for all the setters methods in my class. And a similar readLock() aspect for my getters.

Brian also presented NetKernel. Also awesome. Still trying to get a handle on how exactly we can leverage it for our needs. His Mashup/SemanticWeb/RDF presentation was also good.

Jared Richardson's presentation Shippers Unite was very good. The best part, calling sales people 'Sales Critters'. He had lots of great advice, and great best-practice type information.

Neal Ford's SOA presentation, and his keynote after dinner were also very good. His keynote, "Polyglot Programming" was very interesting. Contrasted the increasing cruft, and complexity (and irrelevancy) of the Java language against the awesomeness of the Java platform. Described the increasing place Dynamic Languages will have within the software industry. In particular their place inside managed runtime environments (either JVM or CLR).

The only major disappointment I experienced was the (apparent) disdain the conference presenters seemed to show for Python/Jython. Lots of Groovy/Ruby/JRuby/Grails rah-rah-rah. I know Rails is the new poster child, and Ruby folks should be proud. But, to give Python/Jython/IronPython hardly a mention. And, the one time the word Python appeared on a slide, to explicitly declare Ruby the winner over Python... seems premature.

Yeah, yeah. Sour grapes. Maybe I should try Ruby before deciding I don't like it... but the Perl-like $ and @ syntax rubs me the wrong way. And, maybe I should just be happy about the rising tide of dynamically typed languages that'll lift all boats.

change the world?

I've been getting more and more excited by the OLPC project. Not only is it exciting technically, it seems like it has huge potential to change the world.

It'll take a few years before the educational goals bear fruit, but it'll have a huge technical impact much sooner. A lot of the cool technology could make its way out to non-OLPC hardware -- displays, power management, networking, security, exciting Python changes... lots of great stuff.

The Daily Python-URL pointed me here. A great introduction to the project, its goals, and the technology.

more IronPython + MarkLogic experimenting

I've been playing more with IronPython. Specifically, experimenting with using it along with the MarkLogic's XCC API.

Pretty ugly, but working re-implementation of the sample code to run queries.

# Re-implementation of SimpleQueryRunner.cs XCC example
# using IronPython

import clr
clr.AddReferenceToFile("MarklogicXcc.dll")

import Marklogic.Xcc
import System
import os
import sys

def execute(session,query):
  """Generator that yields result strings from
  execution of the query"""
  req = session.NewAdhocQuery(query)
  for res in session.SubmitRequest(req).AsStrings():
      yield res
  return

def main():
  if len(sys.argv) != 3:
      print usage()
      sys.exit(2)

  query = "'Hello world'"
  try:
      f = open(sys.argv[2])
      query = f.read()
      f.close()

      cs = Marklogic.Xcc.ContentSourceFactory.NewContentSource(
              System.Uri(sys.argv[1]))
      session = cs.NewSession()

      for result in execute(session,query):
          print result

  except EnvironmentError,e:
      sys.stderr.write("*** Error: %s\n"%e)
      sys.exit(1)
  except Exception,e:
      #ugh, looks like exceptions from XCC are Exception
       sys.stderr.write("*** Error: %s\n"%e)
      sys.exit(1)

def usage():
  return "usage: \n  SimpleQueryRunner.py xcc://USER:PASSWORD@HOST:PORT/MLDATABASE <queryfile>"

if __name__ == "__main__":
  main()

Figuring out more ways I might infiltrate my workplace with Python makes me happy happy. Beer also makes me happy. Mmm... Murphy's Stout.

MarkLogic XCC + IronPython = Sweet!

At work we're using MarkLogic to store and transform our XML content. You use XQuery to access the data, which has been pretty fun to learn. While looking at the release notes for MarkLogic 3.1-2, I saw that they've released the .NET version of their new XCC API for connecting with the server.

I've meant to start playing with IronPython for a while. And I've also meant to start learning C#/.NET stuff. So, I thought I'd try running the MarkLogic XCC.Net examples via IronPthon.

And, it works! Woohoo! Not that it does much yet...

import clr
clr.AddReferenceToFile("MarklogicXcc.dll")

import Marklogic.Xcc

import System

import os

doc = 'hamlet.xml'

#replace connection info and marklogic db name 
contentSource = 'xcc://USER:PASSWORD@HOST:PORT/MLDBNAME'

print "Loading document ..."
Marklogic.Xcc.Examples.ContentLoader.Main(
 System.Array[System.String](
     [contentSource ,doc]
     )
 )

print "Fetching document ..."
Marklogic.Xcc.Examples.ContentFetcher.Main(
 System.Array[System.String](
     [contentSource,os.path.abspath(doc).replace('\\','/'),'-o','hamlet_fetched.xml']
     )
 )
print "Done."

Our collection of XSLT/XQuery/Java applications is a pain to deploy and do quick interactive testing against. Hopefully being able to script it with IronPython -- and maybe provide a nice interface with Windows Forms -- will allow quicker turnaround times.

PythonCard presentation

Last month, I gave a presentation on PythonCard to the utahpython UG . The presentation seemed to go over well.

You can get the source files for the presentation here.