home > developer

Saturday, January 3rd, 2009

Introducing numpyIO.py

I implemented numpyIO.py because I often use scipy.io.numpyio.fread() and scipy.io.numpyio.fwrite(), but I don’t want to depend on scipy. It uses numpy’s tofile/fromfile functions instead.

Why might you want this instead of SciPy’s numpyio? Well, I can think of a few reasons:

  • it imports faster because it only depends on numpy, not scipy
  • easier to package with py2exe or py2app
  • it is slightly faster
  • it lets you avoid rewriting legacy code that makes extensive use of scipy’s deprecated fread/fwrite

Grab the numpyIO project from bitbucket or clone the repo with:

hg clone http://bitbucket.org/lannybroo/numpyio

/developer   〆   permanent link

Monday, December 29th, 2008

Updating to Python 2.6 on Mac OS X Leopard 10.5.6

Install Python from source.

cd ~/tmp
wget http://www.python.org/ftp/python/2.6.1/Python-2.6.1.tgz
untar Python-2.6.1.tgz
cd Python-2.6.1
mate Mac/README
./configure --enable-framework
make
make install
cd ..

Update setuptools so that easy_install works.

wget http://pypi.python.org/packages/2.6/s/setuptools/setuptools-0.6c9-py2.6.egg
sh setuptools-0.6c9-py2.6.egg

Add the following to my .profile so that the binaries installed by easy_install take precedence over my old Python 2.5 stuff.

export PATH=/Library/Frameworks/Python.framework/Versions/2.6/bin:$PATH

Easy install various packages I use frequently.

easy_install IPython
easy_install nose
easy_install pexpect
easy_install cheetah
easy_install clonedigger
easy_install pyserial
easy_install markdown2
easy_install euclid
easy_install psyco

Do the slightly tougher installs: NumPy (svn rev 6271), SciPy (svn rev 5300), matplotlib, and euclid. See http://www.scipy.org/Installing_SciPy/Mac_OS_X for details. Before starting, I also updated to the latest apple developer tools, although this isn’t strictly necessary.

wget http://r.research.att.com/gfortran-4.2.3.dmg
open gfortran-4.2.3.dmg # install using GUI

wget http://www.fftw.org/fftw-3.2.tar.gz
untar fftw-3.2.tar.gz
cd fftw-3.2
./configure
make
sudo make install
cd ..

svn co http://svn.scipy.org/svn/numpy/trunk numpy
cd numpy
python setup.py build
python setup.py install
cd ..

svn co http://svn.scipy.org/svn/scipy/trunk scipy
cd scipy
python setup.py build
python setup.py install
cd ..

python
>>> import numpy
>>> numpy.test('1','10') # 6 problems in 1897 tests, but usable
>>> import scipy
>>> scipy.test('1','10') # 168 errors of 3990 tests, but usable

sudo port install libpng
sudo port install freetype
easy_install matplotlib

ipython -pylab
>>> plot([1,2,3])

svn checkout http://pyeuclid.googlecode.com/svn/trunk/ pyeuclid
cd pyeuclid
python setup.py install
cd ..

Install wxPython and PythonCard for GUI development.

wget http://downloads.sourceforge.net/wxpython/wxPython2.8-osx-unicode-2.8.9.1-universal-py2.6.dmg
wget http://downloads.sourceforge.net/wxpython/wxPython2.8-osx-docs-demos-2.8.9.1-universal-py2.6.dmg
open wxPython2.8-osx-unicode-2.8.9.1-universal-py2.6.dmg
open wxPython2.8-osx-docs-demos-2.8.9.1-universal-py2.6.dmg

wget http://prdownloads.sourceforge.net/pythoncard/PythonCard-0.8.2.tar.gz
untar PythonCard-0.8.2.tar.gz
cd PythonCard-0.8.2
python setup.py install
open /Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/site-packages/PythonCard/samples/minimal/

Then, Ctrl-click and open minimal.py with Python Launcher to test.

/developer   〆   permanent link

Sunday, November 30th, 2008

Speedy Python

A great reference: Tools for Accelerating Python. I learned about quite a few new ways of getting Python code to run faster …

/developer   〆   permanent link

Sunday, November 30th, 2008

Git Link Roundup

I’ve been working more with git lately and have found a few more interesting sites I’d like to share.

The Git Community Book is a great quick reference for finding out how to use a command if you can’t recall from the man page.

If you want to really understand the gory details of branching and merging with git (and I do highly recommend understanding them … easy branching is the best part of git coming from a svn background), then read this long article on LWN.net.

Also, R. Tyler Ballance has a series of blog entries on moving his team from subversion to git. I especially liked Team Development with Git which details the bad habits of developers that are just becoming familiar with distributed source control.

The Battery Powered blog tells us how to deploy a Git Repository Server in Ubuntu, which may prove useful soon. Currently, I’m just using Git for my local work and then pushing to a SVN repo, but I’d like to move others to Git soon.

scie.nti.st also explains how to host a Git repo. Both this and the Battery Powered link above use gitosis to make setup easier.

/developer   〆   permanent link

Monday, September 8th, 2008

Git-SVN Cheat Sheet

Set good “don’t mess with my line endings” option before first checkout:

git config --global core.autocrlf false

Do this one time to suck in the repo w/history from SVN:

git svn clone svn://myrepo.com/project/

Do this loop over and over until your code ready to push back to SVN. The cool thing about this is that the adds and commits can be done off the network! Don’t forget to merge and re-test your changes after the rebase:

git add .
git commit -m "Added some awesome stuff"
git svn rebase

Push all local changes back to the SVN repo, which will now have all the local commits you did when you were on the airplane:

git svn dcommit

One other key tip is what when you want to decommit or rebase but you are in the middle of some changes, don’t worry. git-stash to the rescue:

git stash
git svn rebase
git stash apply

Also, here are a few links that inspired this post:

/developer   〆   permanent link

Saturday, May 10th, 2008

Run Your Software on Your GPU

Check out GPULib from Tech-X Corp. It gives you the ability to run mathematical functions on your GPU card (certain NVIDIA models only, as the moment). Includes a blog and bindings for Python (pystream) and MATLAB.

/developer   〆   permanent link

Tuesday, May 6th, 2008

Idiomatic Python, the Cheese Shop, PyEuclid

Today is becoming python link day, so here are a few more:

  • Idiomatic Python has some great tips on how to use python as it was intended.
  • The Python Package Index is a good place to look for code someone has already written that might solve the problem you’re thinking about, and probably in a much cleaner way than you were thinking. This used to be known as the Cheese Shop.
  • pyeuclid is a nice 2D/3D vector, matrix, and quaternion math library (that needs better docstrings and a ray-sphereoid intersection).

/developer   〆   permanent link

Tuesday, May 6th, 2008

Numpy for MATLAB Users

Great reference for those of us recovering from a MATLAB addiction, like this guy, who recently moved his scientific workflow to Python.

/developer   〆   permanent link

Thursday, May 1st, 2008

Python-by-example

Straight-forward Python reference the explains the standard library rather completely and cleanly just by showing a simple example for each module and function. Very nice for a quick lookup of something you don’t quite remember.

/developer   〆   permanent link

Saturday, October 13th, 2007

effbot

Great Python articles, books and downloads. Visit and enjoy. I especially enjoyed this optimization of a log file parser.

Some other interesting articles:

/developer   〆   permanent link

Thursday, October 11th, 2007

SVN Version in Your Source

Suppose you are using Subversion to manage your code and find that you’d like to include revision information within a file. You might read this and think it won’t work, but do not be dismayed. You really wanted to look at Keyword Substitution in the Subversion book.

There you will find that enabling substution on a file is rather easy. Say you have some code in Spam.py and you’re running TortiseSVN. Then, the basic process is:

  1. Rt click file -> Properties -> Subversion tab -> Properties.
  2. Add a property svn:keywords with value Date Revision Author HeadURL Id
  3. Put a keyword in the file where you want it to expand the thing — I put $Id$ in the docstring of Spam.py. Id is a summary which, after checking in, expanded to "$Id: Spam.py 513 2007-10-10 23:10:30Z username $".

/developer   〆   permanent link

Tuesday, October 9th, 2007

Hex Fiend for Windows

On Mac OS X, Hex Fiend handles editing of binary files beautifully. On occasion, I also need to munge around with large hex files on Windows. Today, I found the XVI32 hex editor — it does the job nicely, letting me delete out segments and re-save the result. It probably wouldn’t work as well with extremely huge files because it does everything in-memory, but for 300MB files, it works great.

/developer   〆   permanent link

Friday, October 5th, 2007

Text Processing in Python

Check out this great book online (there is a print version too).

/developer   〆   permanent link

Monday, September 24th, 2007

Why Rewrites Can Go Wong

Good article series from Chad Fowler on why rewriting software often goes bad if you do it as a Big Rewrite. Kevin Barnes also has some thoughts.

/developer   〆   permanent link

Saturday, April 21st, 2007

Decent Python References

I’ve been doing some more Python development lately. Wanted to document the following decent references online.

/developer   〆   permanent link

Sunday, April 16th, 2006

Language-Maven vs Tool-Maven

In The IDE Divide, Oliver Steele uses his analytical knife to split developers into language mavens versus tool mavens. What would you rather use on your next project?

  1. a powerful language and text editor
  2. whatever language and a powerful IDE

/developer   〆   permanent link

Monday, April 10th, 2006

People Don’t Crunch Well

Evan Robinson gives a clear explanation of why crunch mode doesn’t work.

In most times, places, and industries over the past century, managers who worked their employees this way would have been tagged as incompetent — not just because of the threat they pose to good worker relations, but also because of the risk their mismanagement poses to the company’s productivity and assets. A hundred years of industrial research has proven beyond question that exhausted workers create errors that blow schedules, destroy equipment, create cost overruns, erode product quality, and threaten the bottom line. They are a danger to their projects, their managers, their employers, each other, and themselves.

/developer   〆   permanent link

Sunday, March 19th, 2006

Joel Test for the Web

The Joel Test gives 12 simple tests for the quality of a sofware development team.

Drew Mclellan offers thoughts on The Joel Test for Web Development. He finds that the test is still very useful in the context of web development.

/developer   〆   permanent link

Sunday, March 19th, 2006

Cheat Sheet Roundup

Pete Freitag lists cheat sheets for developers. I love cheat sheets. This covers web, programming, version control, OS commands, and more.

/developer   〆   permanent link

Sunday, March 19th, 2006

Subversion Install and Use on Mac OS X

Justin Williams of MacZealots.com gives some good tips on installing and using subversion on Mac OS X. I found subversion indispensable in keeping track of all the code and writing for my master’s thesis.

/developer   〆   permanent link

Sunday, March 19th, 2006

Haskell for C Programmers

Haskell (a programming language) has no update operator. There is no order of operations. Find out more from this introduction to a “functional” programming language.

/developer   〆   permanent link

Next 16 entries