Wednesday, October 21, 2009

Trials and Tribulations: pyscopg2

So here's my latest debacle that caused me to waste hours of my time trying to find a solution. Let me set the stage:

Actors:
Virtual machine with RedHat ES 4.7
Remote Postgres database

Intent:
Connect to my remote database and list out the contents of a simple table

Sounds easy? I thought it would be but...

I downloaded the latest pyscopg stuff from the site

wget http://initd.org/pub/software/psycopg/PSYCOPG-2-0/psycopg2-2.0.11.tar.gz

and then tried to install and I then get these messages:


python setup.py build
running build
running build_py
running build_ext
error: No such file or directory

After googling with browsers filled to the brim with a plethora of tabs it seems as if I may be missing the development libraries for the package; fair enough lets get those packages.

The version of Redhat that I have installed on the machine seems to be able to support Postgres 7.4 rather than 8.1 so I get the RPMs to support the entire lot.
I would have upgraded to 8.1 just to be current but I would have to install tons of development pakcages to support 8.1 and it just wasn't worth it.


Try to install postgresql-devel-????.rpm and it will point out what other dependencies it needs.

Success! The package compiles and installs. The sun is shining once again and I can almost hear the cherubs singing my praise but lo, a dissenting voice is heard...

Writing the script makes it painfully obvious that we're not done yet.


Importing psycopg from within a Python script, it gives us the error about PQserverVersion not being defined.

After much digging it seems as if the 8.1 libraries (libPQ.so) have the above mentioned function as a new addition and they've also changed the lo_create function to lo_creat so after a little patching...

3. Patch the connection_int.c
change the line:
self->server_version = (int)PQserverVersion(pgconn);
to:
self-server_version = '70401';
4. Patch the lobject_int.c
change the line:
self->oid = lo_create(self->conn->pgconn, new_oid);
to:
self->oid = lo_creat(self->conn->pgconn, new_oid);

After those changes are made and I reinstalled the package, I could import the package with no errors.

I was able to connect to my table at the end of the day but what a path I had to take to get there.

Here's so you don't have to do this bullshit...

Cheers!

Thursday, September 24, 2009

Tix and the Tree Widget

So I'm using Tix for my GUI work and have been exploring the wonderful world of Tix with an introduction to the Tix.Tree

It really is quite simple:


tree = Tix.Tree(master, options='seperator "."')
tree.pack(expand=1, fill=Tix.BOTH, padx=10, pady=10, side=Tix.LEFT)

#fill the tree with our values
tree.hlist.add('one', text='one')
tree.hlist.add('two', text='two')
tree.hlist.add('two.three',text='three')
tree.hlist.add('two.four',text='four')


which will give us this:



Great first pass, lets deconstruct the code:

tree = Tix.Tree(master, options='seperator "."')


The first parameter is the parent window just like all the Tk widgets but options has the seperator keyword which is what is used to parse the strings and create the hierarchical style we are looking for. The following lines are self-explanatory; we are merely creating a list of items where the item 'two' has two other children; two.three and two.four

You tell me, "Pretty simple, dude! But what about the rest of the candy?"
How do we define columns and how do you collapse and expand nodes just like we do in all the file explorers?

First question, how do we set up the columns and headers?

The columns and header information gets passed to the Tree control like so:
Tix.Tree(rootWindow, options='hlist.columns 5 hlist.header 1 seperator "."')

To define them properly we need to digress a little and talk about styles;
styles are a Tix construct that allows you to define a tag which you can impose on any widget's text. This can be defined to include the color, font, padding and other display attributes.
tree.tk.call('tix', 'configure', '-fontset', 'WmDefault')
boldfont=tree.tk.call('tix','option','get','bold_font')
style={}
style['header'] = Tix.DisplayStyle(Tix.TEXT,refwindow=tree.hlist, anchor=Tix.CENTER,padx=8,pady=2,font=boldfont)
The first line is to fix the bug with the windows port of Tix, the subsequent lines are the real meat and potatoes of the style mechanism; we create a dictionary of styles, define a header style with a bold font.

Once this is completed we can define our headers to have the 'header' style; like so:
tree.hlist.header_create(0,itemtype=Tix.TEXT, text='Position',style=style['header'])
tree.hlist.header_create(1,itemtype=Tix.TEXT, text='HostName',style=style['header'])


This takes care of the headers,defining the column width is done like so:
tree.hlist.column_width(0, chars=10)
tree.hlist.column_width(1, chars=20)


The last thing we are interested in is the means to collapse and expand the nodes.This is relatively simple in that the individual nodes and can be defined to have that functionality by calling the setmode on the node entry; if that entry has children, the node will be able to be collapsed or expanded. If the setmode command has not been run on a node with children, it will not have the box icon that controls the expansion/collapse mechanism.

tree.setmode(nodeElement, 'open')


So in the end you can get this:

Wednesday, September 23, 2009

Quest for a GUI

This pythonista loves to do GUI work, but what package to use? What features are we looking for and most importantly for me, how easy is it to install and configure on a target machine?

So, I've been using Tcl/TK with Python and find it suits most of my needs for a simple interface but more and more I'm getting to write applications that are more complex than a few simple modal dialog boxes popping up all over the place.

Tix, which is an extension of Tcl/TK for python gives us more variety in terms of widgets, the widgets I'm most interested right now is the hierarchical list and the tree widget.

I took out Tix for a test drive and right out of the box, there are problems but not so insurmountable that they can't be solved. To begin...

I'm developing with python 2.5 (still supports all the 3rd party tools I have come to know and love) and getting Tix 3.4 to work gave me problems on Windows.

I ran the DemoSHList demo and right away it gave me problems.

The line:


boldfont=hlist.tk.call('tix','option','get','bold_font')


was giving me this annoying error:


_tkinter.TclError: can't read "tixOption(bold_font)": no such element in array


what to do, what to do: It seems as if some Tix stuff is broken in windows (The linux side is another matter entirely!!) and to make a long story short, the fonts and colors are not setup properly for the windows environment.

The fix is relatively simple affair to do, just remind Tix exactly what kind of operating system its running. Just place this line before you call what you think is a system-defined constant.


hlist.tk.call('tix', 'configure', '-fontset', 'WmDefault')


This will set it to the its Window defaults. I would imaging this will be fixed in the next installment but in the meantime, adding the above line won't hurt and will just become redundant in the future.

Running the tree sample, which merely enumerates the contents of my disk worked right out of the box with no changes.

I'll be updating all my Tk code to take advantage of Tix.

I'll keep you posted if I find any other issues in the windows side.

The linux side will have to wait for a bit since my entire customer base is windows based and the only person who uses the tool in linux is myself so I can afford to wait a bit.

Saturday, July 11, 2009

Python Testing (Unittests)

I've never had an aversion to the newest and best ideas out there when it comes to making my coding productive and fun. One idea I've played with for the last few years is Agile development. I've never belonged to an official agile project but I have followed it over the years in blog posts by others and Scott Ambler's musings in Dr Dobbs magazine.

All I can really offer here is what I've done over the past with the python code that I've been developing for the last years and the tools and design patterns I use to make this work for me.So without further ado lets look at some of the tools out there that you will need in order to get this to work for you.

UnitTest & Coverage

A really good tutorial on unit testing that I would have liked to written can be found here and the official Python documentation of pyUnit is definitely a necessary read for additional insights.


These tools are going to form the backbone of your entire developmental system. Test Driven development requires that we create tests first before we even write a single line of production code. This requirement is easily met if we follow the agile development adage that we design only what we need at the present time to fulfill our requirements.

I'm going to stress this even more by offering a trivial example to walk through:

Design a class that has the same functionality as the pushd/popd unix calls

I like to have a unit test per file which makes running the unit test easier from the command line, you can easily comment out sections of your tests and just run the items that are failing without the distraction of everything else blowing up around you as a result of a low-level change you made. To this end I always have the following at the end of all my unittests


if __name__ == '__main__':
unittest.main()


Just run the unittest from the command line
python <nameOfTest>.py


I'm going to pretend that we wrote the following example iteratively by following these requirements or features

  1. class to be instantiated with a directory entry, directory chosen is now the current directory

  2. error to be thrown if directory entry is invalid or non-existent

  3. once class is destroyed, the previous directory it was pointing to is 'pop'ed back


for every element in feature list:
-Write the test,
-Hack the code until it runs the test and passes

Here's the UnitTest:

########################################################################
class testChDir (unittest.TestCase):

#----------------------------------------------------------------------
def setUp(self):
"""Basic setup"""
self.prev
Dir = os.getcwd()
#----------------------------------------------------------------------
def testChDirGood(self):
''' make that we can cd to a directory and pop it '''
targDir = tempfile.mkdtemp()
targObj = chDir(targDir)
self.assertEquals(os.getcwd(), targDir)
targObj = None
self.assertEquals(os.getcwd(), oldDir)
shutil.rmtree(targDir)
#----------------------------------------------------------------------
def testChDirBad(self):
''' change directory to something that doesn't exist'''
oldDir = os.getcwd()
targDir = '/none'
self.assertRaises(OSError, chDir, targDir)
#----------------------------------------------------------------------
def tearDown(self):
"""typical tear down code"""
os.chdir(os.oldDir)



Here's the code:

########################################################################
class chDir:
#----------------------------------------------------------------------
def __init__(self,dir):
"""Push to our directory entry"""
#get the current directory and save it
try:
self.olddir = os.getcwd()
os.chdir(dir)
except (OSError, IOError), msg
raise msg
#----------------------------------------------------------------------
def __del__(self):
"""pop our directory entry"""
try:
os.chdir(self.olddir)
except (OSError, IOError), msg
raise msg


After this iterative exercise, we're left with a unittest that should be exercising all the capabilities of the class in question.

That was a lot of work for a teensy-weensy class that all it does it push and pop directories but the result of all this work will come in handy if I every have to refactor the code. If I need to add capabilities to this code, all I have to do is change the code and run my unittests to determine that I haven't broken anything. Now I'm totally assured that once I run my unit test and it passes, all the code that uses this tested class will behave properly. Thats certainly a load off my mind when it comes to maintaining changes to code; because it allows me to have the confidence to refactor my code and know that it won't create nasty bugs in my application.

How many times have I encountered push back from developers when it comes to changing an already existent feature in an application. They cite concerns about the mountain of work for them to make sure that it doesn't create bugs and regression testing that needs to be done by the QA department for it to get approved. With the existance of unittests these arguments are moot. The regression tests are unittests and they can be run at anytime to gauge the health of your application's codebase. In fact having unittests will allow the developer the freedom of being able to refactor his code with the least amount of deleterious side effects.

Unittests also bring into sharp focus who is the most capable of testing the code one writes; the developer or the QA engineer. If you hear a traditional programmer explain it; QA engineers are not paid the big bucks coders get so it is beneath them to debug their code. All too often we have the phenomenon of throwing code over to the QA department, to have them validate it but is it really QA's job? Who really knows what a particular function call is doing? Who knows the possible problems that can arise from code that is being written but the programmer? A QA engineer will cull the high level bugs but we can't seriously expect a black box tester to somehow have white-box understanding, this seems to be a fallacy that management falls into to appease the programming prima donnas who are just plain lazy, irresponsible and may I daresay stupid and short sighted.

So Unittests are the answer right? Not exactly... we can have unittests up the yin-yang but if they don't test everything about our code, they're not telling us everything we need and worse yet, they're giving us a false sense of security. The answer to this quandary is our next topic:

Coverage!

Friday, May 22, 2009

A picture is worth a thousand words



Just something I came by in my travels 'round the net. After running with some other languages in the past, Python certainly makes me feel like that, unless its the crack that's talking...

Monday, May 11, 2009

Design Patterns in Python

Interesting talk on the merits of Python when it comes to software patterns. I think it would be an interesting exercise to grab one of the popular pattern books and translate the patterns to concrete Python examples and publish that as a programmer's resource. I'd do it but I'm too busy writing this blog.... :)


Sunday, May 10, 2009

Why Python?

A facebook friend, an aspiring programmer was asking me why I think Python is so great. I thought this would be a great way to begin this blog. This would enable me to enumerate what the salient features were for me that make it such a joy to work with.

I'm a code junkie, been that way since I first wrote my first line of code in BASIC, I was always looking for some other language that would address the deficiencies of the previous. I've moved from BASIC, COBOL, Pascal, Assembler, C. Worked with database packages/languages like DBaseV, Clipper, SQLplus. I had to bend my mind from procedural to object-oriented programming (which caused all of us to talk about the objects were were crafting in the first person) which catapulted me into C++ and Windows programming.

Having navigated the previous decade of programming languages and paradigms, which had been a tumultuous time, I'm glad to have discovered the safe harbor of Python. Its great because...and Mark Lutz says it so much better in his book "O'Reilly's Programming Python" and I'll try to paraphrase.

The immediacy of code

Python is an interpreted language. Programs in python consist of modules that are loaded at runtime and their objects executed, because of this one can edit the code while the program is running and have the newly edited code executed when the module is reloaded. Development time decreases dramatically.

You can have a program rewrite its source code, reload and execute it all while its running without skipping a beat, great for genetic programming or having a daemon upgrade itself without restarting it. Other benefits I've found is being able to develop and debug GUI based applications quicker since you don't have to restart the application everytime you change any code. You can edit the code, save the module file and click on the button in your GUI app that will reload the module and run the code. The time savings are enormous and I don't get distracted by waiting for a GUI application to load up.

Pseudocode

Python is a high level language, alot of the low level details are handled by the language itself allowing a simpler language that handles details like dynamic type casting, memory managment and garbage collection. Because all these details are missing we're left with the actual problem that is to be solved.
Reading python programs is much simpler since we're not bogged down with details about memory managment and type casting (in C/C++ they are the major culprits for bugs). Python programs read like pseudo code and makes writing algorithms easier since it allows you to concentrate on the problem at hand and delegate the low level stuff to python.
I've read code that I wrote a year ago (when I was just starting out ) and I was still able to understand it, I wouldn't have been able to do that with C.

Object Oriented

Python allows you to write object oriented code since the language itself is based on objects. All the basic elements are present such as single and multiple inheritance, all that good UML stuff

Hybrid Application development

Python can support calling functions in libraries from other 3rd party languages. All you have to do is create a special python file that operates as a map between the native function and the python function, this lookup is provided to the python interpreter and the call to the library function is made and returned to the python interpreter. This feature allows you to leverage code that has already been written and putting a simpler interface via python. Depending on the application that you're running, the degree of hybridization is determined by performance needs and how easy it is to integrate the 3rd party components.

All these reasons and more have made me a fan of this awesome language and I continue using it with the same enthusiasm from the beginning, so join me as we ride the python and learn something new.