Planet Python/SoC - 2009 edition

James Pruitt: Core Conversion Complete

noreply@blogger.com (Eric) — Thu, 02 Jul 2009 16:58:00 +0000

My file wrapper and Josiah's code now work in Python 3.1 and the changes have been committed to the Google Code project. The next thing to tackle is converting Mark Hammond's code from C++ to C and integrating it into the _subprocess.c file.

Skipper Seabold: Generalized Linear Models

noreply@blogger.com (jseabold) — Thu, 02 Jul 2009 13:31:08 +0000

As I have mentioned, I have spent the last few weeks both in stats books, finding my way around R, and cleaning up and refactoring the code for the generalized linear models in the NiPy models code. I have recently hit a wall in this code, so I am trying to clear out some unposted blog drafts. I intended for this post to introduce the generalized linear models approach to estimation; however the full post will have to wait. For now, I will give an introduction to the theory and then explain where I am with the code.

Generalized linear models was a topic that was completely foreign to me a few weeks ago, but after a little (okay a lot of) reading the approach seems almost natural. I have found the following references useful:

Jeff Gill's Generalized Linear Models: A Unified Approach.

James Hardin and Joseph Hilbe's Generalized Linear Models and Extensions, 2nd edition.

P. McCullagh and John Nelder's Generalized Linear Models, 2nd edition.

The basic point of the generalized linear model is to extend the approach taken in classical linear regression to models that have more complex outcomes but ultimately share the linearity property. In this respect, GLM subsumes classical linear regression, probit and logit analysis, loglinear and multinomial response models, and some models that deal with survival data to name a few.

In my experience, I have found that econometrics is taught in a compartmentalized manner. This makes sense to a certain extent, as different estimators are tailored to particular problems and data. GLM on the other hand allows the use of a common a technique for obtaining parameter estimates so that it can be studied as a single technique rather than as a collection of distinct approaches.

If interested in my ramblings, you can find a draft of my notes as an introduction to GLM here, as Blogger does not support LaTeX... Please note that this is a preliminary and incomplete draft (corrections and clarifications are very welcome). One thing it could definitely use is some clarification by example. However, as I noted, I have run into a bit of a wall trying to extend the binomial family to accept a vector of proportional data, and this is my intended example to walk through the theory and algorithm, so... a subsequent post will have to lay this out once I've got it sorted myself.

Generally speaking, there are two basic algorithms for GLM estimation: one is a maximum likelihood optimization based on Newton's method the other is commonly refered to as iteratively (re)weighted least squares (IRLS or IWLS). Our implementation now only covers IRLS. As will be shown, the algorithm itself is pretty simple. It boils down to regressing the transformed (and updated) outcome variable on the untransformed design matrix weighted by the variance of the transformed observations. This is done until we have convergence of the deviance function (twice the log-likelihood ratio of the current and previous estimates). The problem that I am running into with updating the binomial family to accept proportional data (ie., a vector of pairs (successes, total trials) instead of a vector of 1s and 0s for success or failure) is more mathematical than computational. I have either calculated the variance (and therefore the weights) incorrectly, or I am updating the outcome variable incorrectly. Of course, there's always the remote possibility that my data is not well behaved, but I don't think this is the case here.

More to come...

Skipper Seabold: Project Status

noreply@blogger.com (jseabold) — Thu, 02 Jul 2009 11:45:23 +0000

I have been making slow but steady progress on the NiPy models code. Right now for the midterm review, we have been focusing on design issues including the user interface and refactoring, test coverage/bug fixing, and some extensions for postestimation statistics. Other than this, I have spent the last month or so with anywhere from ten to fifteen stats, econometrics, or numerical linear algebra and optimization texts open on my desk.

The main estimators currently included in the code are generalized least squares, ordinary least squares, weighted least squares, autoregressive AR(p), generalized linear models (with several available distribution families and corresponding link functions), robust linear models, general additive models, and mixed effects models. The test coverage is starting to look pretty good, then there is just squashing the few remaining bugs and improving the postestimation statistics.

Some enhancements have also been made to the code. I have started to include some public domain or appropriately copyrighted datasets for testing purposes that could also be useful for examples and tutorials, so that every usage example doesn't have to start with generating your own random data. I have followed pretty closely to the datasets proposal in the Scikits Learn package.

We have also decided to break from the formula framework that is used in NiPy. It was in flux (being changed to take advantage of SymPy the last I heard) and is intended to be somewhat similar to the formula framework in R. In its place for now, I have written some convenience functions to append a constant to a design matrix or to handle categorical variables for estimation. For the moment, a typical model/estimator is used as

In [1]: from models.regression import OLS

In [2]: from models.datasets.longley.data import load

In [3]: from models.functions import add_constant

In [4]: data = load()

In [5]: data.exog = add_constant(data.exog)

In [6]: model = OLS(data.endog, data.exog)

In [7]: results = model.fit()

In [8]: results.params
Out[8]:
array([ 1.50618723e+01, -3.58191793e-02, -2.02022980e+00,
-1.03322687e+00, -5.11041057e-02, 1.82915146e+03,
-3.48225863e+06])

Barring any unforeseen difficulties, the models code should be available as a standalone package shortly after the midterm evaluation rapidly approaching in ten days. The second half of the summer will then be focused on optimizing the code, finalizing design issues, extending the models, and writing good documentation and tutorials so that the code can be included in SciPy!

Priit Laes: Basic PDE separation of variables implemented

Thu, 02 Jul 2009 03:53:45 +0000

For the most of the past week, I have been working on variable separation of PDE's and yesterday I sent my merge request. Currently two mostly-used strategies - additive and multiplicative separation are implemented and tested on simpler equations like Laplace's.

So, what's next? :)

I am currently prototyping a pdesolve() which can handle first-order equations (and also any hyperbolic PDEs) using characteristic methods. Once this is finished, I can start adding the variable separation into pdesolve() and after this has been done, I can move on to implment support for boundary and initial conditions.

Danilo Freitas: new and del operators working!

dsurviver@gmail.com (Danilo Freitas) — Thu, 02 Jul 2009 01:07:17 +0000

Now we can instanciate C++ objects dynamically on Cython.
thins like
cdef Foo *aFoo = new Foo()
del aFoo
are allowed.
We're using Python Syntax for it, so, we define the contructor with "__init__" special method.
Now, the next step is to work with function overloading. We already have the logic to implement it. So, what I need now is to work :).
But, before it, I shall write some tests and some documentation, and my branch of GSoC will be merged to the main branch of Cython, and so, let users do what is needed.

Skipper Seabold: Econometrics with Python

noreply@blogger.com (jseabold) — Wed, 01 Jul 2009 22:03:41 +0000

There is as yet no equivalent of R in applied econometrics. Therefore, the econometric community can still decide to go along the Python path.

That is Drs. Christine Choirat and Raffello Seri writing in the April issue of the Journal of Applied Econometrics. They have been kind enough to provide me with an ungated copy of their review, "Econometrics with Python." Mentioning the, quite frankly, redundant general programming functions and tools that had to be implemented for R, the authors make a nice case for Python as the programming language of choice for applied econometrics. The article provides a quick overview of some of the advantages of using Python and its many built-in libraries, extensions, and tools, gives some speed comparisons, and also mentions a few of the many tools out there in Python community for econometrics including RPy (RPy2 is now available), and of course NumPy and SciPy. Having spent the last week or more trying to master the basic syntax and usage of R, I very much sympathize with this position. The one complaint I hear most often from my fellow students is that Python is not an industry standard. I hope this can change and is changing, because it's much more of a pleasure to work with Python than the alternatives and that makes for increased productivity.

Danilo Freitas: Some questions about syntax

dsurviver@gmail.com (Danilo Freitas) — Wed, 01 Jul 2009 13:34:30 +0000

We are deciding what syntax to use on operators overloading and constructor.
We have C++ and Python syntax, and we have big differences between them.
A thread was created on cython-dev list for voting and commenting the choose of the syntax.
On C++ syntax, the constructor is a method with no return type (it's not void!) and the operators are methods with the keyword "operator" followed by the operator to overload (e. g. "bool operator==(int value);").
On Python, we have the special method __init__ for constructor and other special methods for the operators (__add__, __str__, etc.)

For more information, take a look at the wiki of the project:
http://wiki.cython.org/gsoc09/daniloaf/progress

James Pruitt: Not Far From The Tree

noreply@blogger.com (Eric) — Tue, 30 Jun 2009 18:30:00 +0000

Now that my implementation of subprocess.Popen is complete and unit tested, I will be moving on to moving my changes to Python 3.1. I just compiled Python 3.1 on Debian, all modules included, and pdb is still mal-functioning so if the code conversion isn't as smooth as I anticipate, I will be using print statements to debug which is a bit difficult due to the nature of subprocess.Popen.

The license of my project has been changed from GPL to Apache License 2.0 so that there is a chance of it being integrated into the Pytohn core. There are still some issues that I may have handled incorrectly as far as my subprocess.Popen file wrapper goes, mainly handling the "mode" argument that is used when one opens a file. Right now, it is mostly ignored, the exception being universal newline support.

Dale Peterson: Code cleanup and updating of examples

Tue, 30 Jun 2009 14:35:38 +0000

A few days ago, I discovered the .get() method of the dictionary data type. It allows you to request the value corresponding to a key which a dictionary may not have, in which case you can specify what should be returned instead of rasing a KeyError. In PyDy, I had numerous cases where I was using if/else statements to check if a dictionary had a certain value, but by using the .get() method, I was able to eliminate about 50 or 60 lines of code and make the code a lot more readable and easier to maintain.

The progress on the trigonometric functions has been slow. Ondrej and I worked on some code together that handles most of the cases in the table by Fu et al., but not all. One of the things that has been a little challenging is to identify pi shifts in the argument to the trig functions and map the shift into the first quadrant (0, pi/2) and return the appropriate result, e.g.: sin(x+17*pi/18) –> sin(x + pi/18), etc. The approach has been to match x and r in the following generic argument:

x + r*pi

As I see it, there are three cases to consider: 1) x==0, r!=0, 2) r==0, x!=0, and 3) x!=0, and r!=0. I am splitting the logic up in this fashion and am trying to deal with each case in the appropriate manner. In most cases, r will be of the Rational class, and therefore the modulo operator is needed to bring it to the interval (0, 2*pi), so I implemented the necessary code and sent a patch to the Sympy mailing.

Although the trig functions aren’t quite as they need to be yet, I was still able to manually get the simplifications that are presented by Fu et al. to work using the builtin Sympy commands, which hopefully will mean that simplifying the implementing the algorithm will be feasible with the existing Sympy tools and not too many extra functions will need to be created.

Kang Zhang: Porting to Gnome Keyring

Jobo.ZH@gmail.com (Kang) — Tue, 30 Jun 2009 13:21:00 +0000

I've finished porting the API to Gnome keyring. The latest code on the repository includes this feature. To compile the lib for gnome-keyring, you need Glibc 2.0, D-Bus 1.0 and Gnome Keyring 1.0. The changes that the keyring api made on your os can be see in the Seahorse or Gnome Keyring Manager( Deprecated).

Next platform is KDE Kwallet, here's some interesting materials .

M. Shuaib Khan: Progress...

noreply@blogger.com (M. Shuaib Khan) — Tue, 30 Jun 2009 13:57:33 +0000

I've had little progress this week, and couldn't update my blog last week, due to final exams of my final semester. But since the exams are about to end, I think at least I can document my previous to last week's progress.

I analyzed a good number of python C modules using gcov manually. The results seem good and encouraging and gcov seems to be the right tool for the coverage. The fact that it requires modules to be linked in statically into Python is a bit of a trouble, but after having a detail discussion with my supervisor, Seth, we decided on assuming the following two points for my later work on integrating the gcov reports into figleaf:

The user has already compiled Python with modules statically linked in
Proper arguments were used with gcc during the compilation process so to let gcov track the coverage.

Lots of manual work during the past weeks. It is actually time to start coding all of it, and I've to start first with letting figleaf take care of the C code coverage report generation too. I've been looking through the figleaf code to figure out how to best achieve that goal, and will document any changes made here on the blog.

In the meantime, I've also started to look into improving the test coverage by writing new test cases. I found the unittest module to be interesting to start with as it has around half coverage, and would be interesting writing unit tests for the unittest module itself.

Fabian Pedregosa: Preparing a new release

Tue, 30 Jun 2009 06:57:48 +0000

Last days I’ve been busy preparing the first public beta of SymPy 0.6.5. Most of the time was spent solving a bug that made documentation tests fail under python2.4, but now that this is solved, I hope that by the end of the week we could have a final release.

When this release is published, we’ll merge my query module and work on getting it right for 0.7.

Tyler Laing: Movie Module Progress: Week 6

Tue, 30 Jun 2009 01:58:49 +0000

So, week six. I devoted this entire week to exhaustive testing and bug fixing. Its just good software development to take some time, and test and fix bugs. No new features, nothing. I also took the time in the process to refactor a section of code I felt was extremely hacky and brittle. The benefit [...]

Freddie Witherden: x Marks the Spot

noreply@blogger.com (Freddie Witherden) — Mon, 29 Jun 2009 19:56:44 +0000

There it is — the first 'equation' ever rendered by Mathtex! Although it may look like nothing more than a 99-DPI 12pt x in italicized Computer Modern it is really something quite special — a vision of progress.

Below is the parse-tree and glyph stream generated by the program:
freddie@fluorine ~/Programming/mathtex $ python main.py
[Hlist 9.42> [Hlist 0.00> ] [Hlist 9.42> `x` k1.17] [Hlist 0.00> ]]
[(-0.5, 7.0, Bunch(symbol_name=x, metrics=Bunch(advance=9.41821289062, iceberg=7.0, ymax=7.0, height=7.0, width=8.25, slanted=True, xmax=8.6875, xmin=0.5, ymin=0.0), num=120, fontsize=12, offset=0.0, postscript_name=Cmmi10, font=, glyph=))]

Over the last couple of days I have been working on the code that I committed last week (here for those that are interested) and as promised now have something that does work.

However, there are several unpleasantness associated with it: firstly it depends on mathtex.ft2font — the FreeType wrapper used by matplotlib; secondly there is currently only a Cairo backend; thirdly the only font series supported Computer Modern, by way of the Bakoma fonts; fourthly the font paths are currently hard-coded.

I plan to fix all of these issues over the next couple of days — starting with using font metrics files as opposed to FT2Font and then writing a C-based renderer and wrapping it using Cython. I expect that this will be done by Friday.

Vicent Marti: Gentlemen. An update.

Sat, 27 Jun 2009 20:05:00 +0000

Hey everybody. Apologies beforehand for my lack of updates -- I know I'm totally failing at maintaining this blog (as usual, and as expected), but that doesn't mean that progress is not being made.

Far from that, actually. I've finished my exams break last week and resumed my usual schedule; everything is looking very in shape for the first evaluation on July the 6th. Let's take a look at the SVN logs and see what has been done... After the break.

::READ MORE

James Pruitt: Updates

noreply@blogger.com (Eric) — Sat, 27 Jun 2009 14:52:00 +0000

Though it has been a while since I have made a post, things are going well right now. Since my last post, I have created a Google Code Repository located at http://code.google.com/p/subprocdev/. I have integrated, modified and written tests for the code I got from Josiah Carlson. Two of the functions he left outside of his Popen patch that were moved into subprocess.Popen. I made some aesthetic and functional changes to his recv_some and send_all functions.

Eventually, I got my code and Josiah's code just about fully unit tested in Python 2.7 and from there, proceeded to see if I could convert it over to 3.0. After I did this, I realized that it still ran, for the most part, in Python 2.7. With the following code, I attempted to get around the only thing that was causing me grief:

import sys
if sys.version_info[0] == 3:
from io import BufferedWriter as buffer
else:
class str(object):
def __init__(self,a,b=None):
self = a.__str__()
def isinstance(a,b):
if b == str:
return hasattr(a,'strip')
try:
return a == b(a)
except:
return False

In Python3.0, the str function can also accept character encoding as an argument so I tried to overload the Python 2.7 str class to make it 3.0 compliant to no avail so I will be producing a separate branch for Python 2.7 and 3.0 development in my Google Code Repository but any suggestions on getting around the str issue are greatly appreciated.

Yang Yang: Porting py.test from python2.x to python3.1 (updated on Jun 27)

yangyan5@gmail.com (Yang Yang) — Sat, 27 Jun 2009 12:10:48 +0000

Due to python3.x 's intentional backwards incompatibility, it is not an easy work to migrate a project's codebase from python2.x to python3.x. Guido has given a recommended development model:

You should have excellent unit tests with close to full coverage.
Port your project to Python2.6.
Turn on the Py3k warnings mode.
Test and edit until no warnings remain.
Use the 2to3 tool to convert this source code to 3.0 syntax.
Test the converted source code under 3.0.
If problems are found, make corrections to the 2.6 version of the source code and go back to step 4.
When it's time to release, release separate 2.6 and 3.0 tarballs.

Fortunately, py.test is already 2.4 and 2.6 compatible, and has sufficient unit tests. So, I can directly start from step 3. However, unfortunately, Holger requires the codebase ported to 3.1 should still be compatible with 2.4 and 2.6, i.e., maintain only one codebase which could be used through all main python versions. So, 2to3 tool can only be used to show the minimum places that need notification.

Thus, to make py.test compatible with python2.4, 2.6, and 3.1, the first thing is to write some wrapper functions. The functions I wrote are:
Print:
--use Print function instead of each print statement in py.test, so python3.1 will not throw SyntaxError. In Print function, execute correct print code according to current python version.
Raise:
--the only incompatible syntax is raise cls, value, tb (2.x) and raise cls(value).with_traceback(tb) (3.x). So, just simply call corresponding statement in Raise function.
isinstancemethod, isclassmethod, isfunction:
--suppose we defined a class "myclass" and a method "method" inside "myclass", then create an instance of myclass, myinstance. In 2.x, myclass.method and myinstance.method have the same attribute names. But in 3.x, myclass.method's attribute names are the same with the ones of normal function. This is a very annoying difference, because in many places code objects are got by obj.im_func.func_code. In 2.x, obj could be myinstance.method or myclass.method; In 3.x, obj.im_func.func_code must be changed to obj.__func__.__code__ and it only works for myinstance.method. For myclass.method or normal function, should use obj.__code__. So, I have to provide three functions to distinguish them respectively.
updatemethodattr, updatefunctionattr:
--method's attributes:
im_self ==> __self__
im_func ==> __func__
im_class ==> disappeared?
--function's attributes:
func_closure ==> __closure__
func_code ==> __code__
func_defaults ==> __defaults__
func_dict ==> __dict__
func_doc ==> __doc__
func_globals ==> __globals__
func_name ==> __name__
So, to keep codebase unchanged, if obj is 3.x's method or function, create 2.x's attributes.
CmpToKey:
--in 3.x, there is no cmp keyword in sort function. So, this wrapper will transform a cmp function to key function.
bytestostr, strtobytes:
--in 3.x, all strings are unicode, but lots of streams require bytes object instead of string. So, use these two functions to wrap stream arguments and when it's 3.x, transfer from bytes to str or str to bytes.

Second, for some well known incompatibilities, such as "except as" and module rename. For "except Error as e:", change it to "except Error:", and in except block, create e by "e = sys.exc_info()[1]". For module rename, add a try/except block. For example:
try: import StringIO
except ImportError: import io as StringIO

Third, 3.x has many incompatible mechanisms. For example: 3.x will not call __cmp__ when compareing two objects. So, __lt__, __gt__, __le__, __ge__, and __eq__ should be implemented. Furthermore, if you define __eq__, __hash__ must be provided.
In 2.x, dict.items() will return a copy list of key-value pairs. But in 3.x, dict.items() will return a view object, and if the dict's size changed during iteration, a RuntimeError will be thrown. So, although "for key,value in dict.items():" still works in 3.x, it should be changed to "for key, value in list(dict.items()):" in case of RuntimeError.

There are still lots of incompatibities need to be fixed. Currently the packages of py.test without test failures under 3.1 are:
builtin/
cmdline/
io/
log/
path/
process/
rest/
I will keep updating this post when porting. Hope this could be finished ASAP.

Freddie Witherden: This Week in GSoC Mathtex

noreply@blogger.com (Freddie Witherden) — Sat, 27 Jun 2009 10:49:22 +0000

Officially, according to my schedule this week should've been spent on producing a set of unit tests. Now, usually when things don't go to schedule it is because something bad or unexpected occurred.

However, while looking through the Mathtex code last week something good — but unexpected — occurred. It seems as if splitting Mathtex from Matplotlib is significantly easier than I first anticipated. Therefore, this week has been spent splitting the behemoth mathtex.py file in Matplotlib into several smaller files, ready for externalisation.

I expect that by Sunday or Monday the SVN repository will have a version of mathtex that is able to render equations using a Cairo backend and the FT2Font library from Matplotlib. Once this is working it shouldn't be to difficult to a) add a bitmap backend using FreeType/Cython/libpng; b) use font metrics files as opposed to FT2Font for metrics information.

On a personal note yesterday was also my last day in university accommodation. As of 22:30 BST I am now home again as opposed to being in central London. Yay for packing and unpacking!

Priit Laes: Separation of variables for PDEs

Fri, 26 Jun 2009 12:01:31 +0000

School finally ended last week (no exam results yet, though) and I took a long-long trip back home.. Week started slowly due to the deadly combination of the forgotten laptop power cord, Midsummer day, Victory Day and best weather ever. This resulted in lots of grilling, beer and also a small side-effect: my GSoC mentor is probably quite angry now for missing my weekly update again...

Back to Sympy now.

So far all my various doc-related fixes have been committed to upstream, and I have decided to bite the bullet and start implementing the variable separation for PDEs. So far some very simple testcases based on first-order equations work, but I have some trouble with equations containing higher order derivatives. You can pull from gihtub/pde-separate and please break it :)

Freddie Witherden: We Have a Mailing List

noreply@blogger.com (Freddie Witherden) — Thu, 25 Jun 2009 11:57:21 +0000

Following on from the Mathtex project announcement last week we now also have a mailing list. mathtex-dev; http://groups.google.com/group/mathtex-dev?lnk= which is open to all. Although a development list anyone with an interest in the project, should make their voices heard. This will almost certainly become more important in the next week or so when the floor is opened to feature/enhancement requests (backends and syntax support).

Tyler Laing: Science Fiction predicts the future?

Thu, 25 Jun 2009 02:33:38 +0000

I’m watching a documentary on the parallels between the market crash of 1929, and the market crash in 2008(last year). I was reminded of the exploits of Adam Zimmerman, of Brian Stableford’s Emortality series, where Adam used his brilliance in the stock markets to cause a massive market crash, while the masters of the business [...]

Eden Elos: Status update

noreply@blogger.com (Eden Elos) — Tue, 23 Jun 2009 22:47:13 +0000

This blog will keep tracking the development of Interfacing pygr with Clustalw alignment, BLASTZ alignment, BLAT alignment, LAGAN alignment, MLAGAN alignment, and gmap alignment. The Project is part of Google summer of Code 2009, and will be mentored by Dr. Titus Brown and Jenny Qing Qian , to both of whom I am very grateful.
Currently, Pygr is interfaced only with BLAST alignment program.

For the actual interfaces, I will write separate Python modules (by the way, Python is the language I am using). Before mid-term evaluation (which is on July 06, 2009), I plan to finish the first three alignment programs and the remaining three will be finished after the mid-term. So far I have finished interfacing Clustalw and blastz with pygr.
The mainstay of this blog will be progress reports. Comments are welcome.

Tyler Laing: Intelligence Gathering Issues

Tue, 23 Jun 2009 18:09:17 +0000

All too often we see reporters miss the real questions or lob easy ones at the Powers That Be. This isn’t enough to make me paranoid, as generally, people do good. But one particular line in this Time article about missing boy Danny Barter really got me interested. The line in question is this: [...]

Danilo Freitas: new and delete operators

dsurviver@gmail.com (Danilo Freitas) — Tue, 23 Jun 2009 14:59:23 +0000

last week we begun with the work to support the 'new' and 'delete' operators.
A new node was created (NewStatNode) and a function in the parser to call it (p_new_statement).
I hope everything continue with no problems.

The midterm evaluations is coming. I hope everything be on time until there. :)

Dale Peterson: Still working on trigonometric.py

Mon, 22 Jun 2009 23:30:15 +0000

I am still working on modifying trigonometric.py to make all of the trig methods behave as they would in Mathematica/Matlab/Maple. This involves getting the .eval method of each of them correct, and then from there I can work on implmenting a new trigsimp.

Fabian Pedregosa: Queries and performance

Mon, 22 Jun 2009 22:02:22 +0000

After some hacking on the queries module, I finally got it right without the limitations of past versions. You can check it out from my repo http://fseoane.net/git/sympy.git, branch master.

It now relies even more on logic.inference.satisfiable(), which is just an implementation of the DPLL algorithm. Bad news is that (my implementation of ) dpll_satisfiable() is SLOW, so inevitably queries are SLOW. But everything is not lost, since the algorithm is quite fast, and in fact other variants of the algorithm (MiniSAT) perform 6600x times faster than my implementation on medium-sized problems (60 variables, 170 clauses). So this looks like something smells bad on the programming side …

However, I spent the day profiling the function (link to source code used for profiling) without much success

Werner Laurensse: GSOC Week 3-4

Mon, 22 Jun 2009 15:19:35 +0000

The past two weeks I was busy studying for my examinations, so I didn’t have much time to work on my GSOC project. However I was able to do a lot of work to get the PyCameraObject struct integrated in the existing code and implemented some additional functions.

Here is the definition of this struct, it isn’t completed yet because some “types” I would like to put into it cause compile errors.

PyCameraObject:

typedef struct {
    PyObject_HEAD
    char* device_name;              // unieke name of the device
    SeqGrabComponent component;     // A type used by the Sequence Grabber API
    SGChannel channel;              // Channel of the Sequence Grabber
    GWorldPtr gWorld;               // Pointer to the struct that holds the data of the captured image
    Rect boundsRect;                // bounds of the image frame
} PyCameraObject;

In addition of implementing the PyCameraObject struct, I have writen the folowing functions.

mac_device_init():
This function initializes the sequence grabber component and prepares it for capturing images.
mac_device_close():
I have extended this function so it works with the PyCameraObject struct.
mac_start_capturing():
At this moment this function
camera_start():
This function just calls the above functions.
camera_get_size():
Simple function that returns the size of the image frame.

Schedule for this week

This week I am planning to finish the mac_start_capturing function so that I can capture images from my camera an put then in a SDL_Surface object. As usional you can folow my work on my github account.

Andrew Friedley

noreply@blogger.com (Andrew Friedley) — Mon, 22 Jun 2009 14:58:45 +0000

This past week I wrote code for int32 and float64 types, giving me an addition ufunc that supports 32/64bit ints and floats. The code for each case is pretty long; each datatype duplicates code that loads the function arguments, checks for cases like reduction and lengths that don't evenly divide the SIMD parallelization/loop unrolling, etc. So I started factoring this out and developing the beginnings of a framework for writing CorePy-based ufuncs with minimal effort and code duplication.

I succeeded in isolating most of the duplicate code, though the 'generic' code generation is a bit more complicated than before -- especially the specialized reduction code. This needs some polish still; I used two different approaches for factoring out the operation-specific code for general element-wise operations and reductions. I will probably use the approach I did for reduction -- the generic code requires a function (I can actually pass a single instruction) that takes two registers as arguments (accumulator and source element) and generates reduction code for a single element.

I ran into a few problems while working on this stuff; I have no idea how I managed to miss these bugs until now. First, applying my ufunc to a single-element array gave bad results -- the test I was using to detect reduction was also catching this single-element case, and not working properly.

The other issue I've run into is due to the way I'm splitting work among multiple threads. I just did the easiest thing possible (halfway expecting to do something more elaborate later) -- divide the length evenly among each thread, giving the last thread additional work if the number of threads does not divide the length evenly. The problem with this is that individual threads will get work segments with unaligned memory if the work segment length is not a multiple of 16 bytes. So this problem is the next thing I need to tackle -- I've been thinking about ways to split the work evenly, while also rounding work segments to a multiple of 16 bytes.

Aaron Meurer: asmeurer

Mon, 22 Jun 2009 05:16:23 +0000

So I pushed some changes to github so Ondrej could help me debug the nseries tests, when I noticed that the changes that I pushed had some bad comments. So I decided to rebase. But git rebase -i told me that there was already a rebase in progress. I figured that I must have done it a long time ago and forgot to abort, so I ran git rebase –abort.

DON’T DO THAT.

I noticed my editor was telling me that an open file had changed. Then, I noticed that ALL of my uncommited changes were gone! And, being uncommited changes, git did not have them saved anywhere!

So now I started to panic. I had done a lot of work on dsolve that I hadn’t commited yet. Normally, I have hourly backups run by Time Machine, but I am on vacation and my backup drive is at home. So I started to see if I could retrieve it somewhere. grep quickly told me that it wasn’t in the hidden git directory, but it was still in my .pyc files. But a Google search told me that retrieving from that is not so easy, if not impossible with Python 2.6. So then, I decided to see if there was any lingering stuff in my virtual memory from my editor. So I ran grep on my harddrive and waited.

While I was waiting, though, I noticed when I scrolled up in my command history that my lost changes were in my Terminal. It turns out that I had just run git commit –interactive and had used * on my patches, so it gave me everything!

So I copied my Terminal history and will work on putting everything back tomorrow. It should be easy, assuming that git apply works for the format that git gives in commit –interactive.

So the lessons are: Don’t abort a rebase without commiting. Don’t start a rebase and then leave it there. Look in your Terminal history if you loose stuff. And it might be a good idea to make manual backups if you are away from your backup drive for a while.

This also highlights why it is important to try to recover data immediately after realizing that it is gone. If I had closed my Terminal session or filled it past the maximum number of lines, my data would be gone. Even if it were in my virtual memory, that wouldn’t last forever either.

Wojciech Walczak: gminick

Sun, 21 Jun 2009 12:54:00 +0000

At the moment the directory structure of my project built upon Sphinx’s dir structure looks like this:

builders/webapp/ -> webapp builder used by Sphinx to build the docs and the webapp,
builders/webapp/templates/ -> the *.py template files rendered by webapp builder (at the moment one file – server.py – lies here, but actually this file is simply copied, not rendered, but it might change in the future),
builders/webapp/templates/html/ -> the templates of HTML code rendered by the middleware (imported by server.py) when serving the docs,

web/middleware/ -> middleware for Xapian, OpenID and serving html files in general,

themes/basic/web/ -> templates for full HTML files. These files are used as templates twice. First, when Sphinx is building the docs, it extends these files with “layout.html”, and then, for a second time, when the files are accessed through the browser. That’s why these files are separated from those located in builders/webapp/templates/html/ directory, which are rendered only once, by the middleware.

A bit of mess, isn’t it? Comments are appreciated

The structure for generated docs and webapp looks like this (to be found in _build/webapp):

html/ -> files from builders/webapp/templates/html and themes/basic/web directories are located here. We don’t have to separate them anymore, because they are all used as templates by middleware now.

openidstore/ -> OpenID directory. It is created and administered by python-openid library.

public/ -> files built by Sphinx go here. This is the directory which should be served

xapian_db/ -> Xapian’s database.

server.py -> a script which imports the middleware and serves the docs. The idea is that: by adapting the code from this file to one’s framework, one should be able to use this webapp in his python web framework of choice. I plan to change it soon, so all server.py logic will be available as a middleware, because the middleware (Xapian, OpenID, etc.) pieces are actually unseparable, so there is no need for an ordinary user to care about importing them all.
Nevertheless, that’s how it looks like now. It may all change in the future, but if we are about to discuss it, it’s good to describe it as it is first.

Aaron Meurer: asmeurer

Sun, 21 Jun 2009 04:48:54 +0000

So I have spent the past week refactoring expand so that you can have more control over what expansion methods you use. With the present method, expand takes in hints which defaults to basic. basic distributed multiplication over addition (), expanded multinomial expressions (), expanded logarithms ( and ), and expanded powers ( and ).

If you wanted to do any of these things, you had to use expand_basic, which did all of them. Also, you had no control on how deep the expansion went. It went all they way down in recursion, so if you only wanted, for example, to distribute multiplication on the top level, it was impossible.

So I decided to start and fix issue 1455. I now have a branch ready in my github account (see here).

With my patch, you can now choose to expand using each of the above individually with the log, mul, multinomial, power_exp, power_base. In addition to this, you also now have complete control of how deep the expression recurses in the expand. Previously, you couldn’t, for example, expand to . (It would also distribute the exponent, then expand to ). Now, you can choose to only distribute multiplication over addition, and to only do it on the top level.

Automatic expansion of exponents
I mentioned above that expand would expand to . Actually, in the current version of SymPy, this would not happen because it automatically combines exponents like to .

I have been working for the past few weeks changing this as per issue 252. I have been mostly successful, except there are two nseries tests that I cannot figure out how to fix. If you think you know why nseries would fail without automatic combining of exponents, please let me know. The expand branch in my GitHub repo also has the exponent patches in it, if you want to see what I mean.

Because some things, like the gruntz algorithm, rely on automatic combining of exponents, so I had to rework powsimp, which combined exponents but also combined bases ( to ) so that it could only combine exponents. That way, I could use it to get the old behavior where I needed it. Use powsimp(expr, deep=True, combine='exp') in my branch to get the old automatic behavior.

So what does this have to do with ODEs?
When I started working on separable equations, I wanted to be able to separate things like into , which is separable in and . So that is why I needed to refactor expand (I don’t, for example, want to change to because the later is not easily recognizable as separate. Doing this or course required that SymPy didn’t automatically put back together into , so I had to fix that issue as well.

I am almost finished implementing separable (there are some match issues that I will blog about later whenever I get them straightened out), and the only thing that is holding all of this back is those nseries tests. If anyone is familiar with how those algorithms work and which parts require automatic combining of exponents, that would be great.

Fabian Pedregosa: Reading CNF files

Sat, 20 Jun 2009 14:27:27 +0000

The DIMACS CNF file format is used to define a Boolean expression, written in conjunctive normal form, that may be used as an example of the satisfiability problem.

The new logic module (sympy.logic) can read the content of a cnf file and transform it into a boolean expression suitable for use in other methods.

For example, let quinn.cnf be a file with the following content:

c  an example from Quinn's text, 16 variables and 18 clauses.
c Resolution: SATISFIABLE
c
p cnf 16 18
  1    2  0
 -2   -4  0
  3    4  0
 -4   -5  0
  5   -6  0
  6   -7  0
  6    7  0
  7  -16  0
  8   -9  0
 -8  -14  0
  9   10  0
  9  -10  0
-10  -11  0
 10   12  0
 11   12  0
 13   14  0
 14  -15  0
 15   16  0

Then we can load the file and test for satisfiability:

In [1]: from sympy.logic.utilities.dimacs import load_file

In [2]: expr = load_file("quinn.cnf")

In [3]: from sympy.logic.inference import satisfiable

In [4]: satisfiable(expr)
Out[4]:
{cnf_1: True, cnf_11: False, cnf_12: True, cnf_13: True, cnf_14: False, cnf_15: False, cnf_16: True,
 cnf_2: False, cnf_3: True, cnf_4: False, cnf_5: True, cnf_6: True, cnf_7: True, cnf_8: True, cnf_9:
 True}

References: more on the DIMACS CNF file format
BUGS: For large files like this one it exits prematurely with an error in the DPLL algorithm.

Tyler Laing: Movie Module Progress: Week 5

Fri, 19 Jun 2009 21:40:44 +0000

So this is it, week 5 of 12/13 weeks. Almost halfway through the summer of code! So, I managed to get sound working properly, with only minimal skips of quality issues. I fixed the buffer issues I mentioned before, so sound sounds great. As well, I worked on several other issues within the source code, various bugs, [...]

Freddie Witherden: We Have a Project!

noreply@blogger.com (Freddie Witherden) — Fri, 19 Jun 2009 20:03:27 +0000

Today I got around to creating a Google Code project for Mathtex in preparation for the pending source code commits. The project can be found here: http://code.google.com/p/mathtex/

Don't expect the front page to say like that for long, however!

Fabian Pedregosa: Logic module merged

Fri, 19 Jun 2009 10:23:41 +0000

Yesterday I finally merged the logic module in sympy’s official master branch, and should be released together with SymPy 0.6.5.

Next thing to do: profile the code and write some docs before the release.

Freddie Witherden: Rendering and Backends

noreply@blogger.com (Freddie Witherden) — Thu, 18 Jun 2009 20:17:56 +0000

As well as familiarising myself with the existing Mathtex code I have also been considering how to actually go about rendering parsed expressions. Matplotlib currently has a very rich collection of backends which all provide significantly more functionality than required by Mathtex — which only needs glyph setting and line drawing.

One of the outstanding issues, however, is that of FreeType. Although there are bindings for virtually every other programming language there are currently none for Python. (There are, however, two failed attempts.) As a result of this Matplotlib includes its own wrapper FT2Font which is C++ based.

In the current implementation of Mathtex all backends use this FreeType wrapper to get glyph metrics (width, height, advance, etc). Some backends then go on to use FreeType for the rendering, while others (such as PDF and SVG) do not. As glyph metrics are, for the most part, invariant I have been considering putting them in a table as opposed to reading them from the font file each time. This is similar to how TeX operates.

The immediate consequence of this would be that FreeType would not be a hard dependency — if one wishes to only produce PDF/SVG files (or use backends which use FreeType indirectly). However, it might lead to reduced rendering quality for bitmapped backends. (I am still investigating this; the reason being that FreeType provides hinted metrics, while a table would not.)

Assuming that there is no perceivable difference then it is likely that for most of the default fonts a look-up table will be produced. This will allow for separation of the parsing/rendering stages: one piece of code can parse the expression and produce a list of glyphs (at specific sizes, styles and locations) and another independent piece of code can then go onto render it. Furthermore this would make it easy for people to make use of Mathtex in their own applications, but just asking for a stream of glyphs/drawing ops and then rendering it themselves.

For the bitmapped backend (arguably the most common) it is likely to be written in C and abstracting away all of the FreeType/compositing operations — which are more natural in C than Python — and then using Cython to create a Python wrapper for it.

If both of these prove viable then they will serve the purpose of sidestepping the FreeType + Python issue entirely. Of course, I think most agree that a well maintained FreeType wrapper for Python is the way to go (and is something I will consider if I have time at the end of my project).

Otherwise my current plan is to use FT2Font from Matplotlib as a wrapper around FreeType. Expect an answer in the next couple of days :)

Kurt Smith: freemalloc

Thu, 18 Jun 2009 18:34:32 +0000

I’m glad to say that the f2cy part of the GSoC miterm evaluation is functional. By that I mean it can wrap functions & subroutine subprograms (not in modules, yet) with any integer or real argument. The testing framework is all in place, and all tests pass. There are still many things that I’d like to add before the midterm, but the basics are all there, and the additions should be fairly modular and easy to do. The code could use some cleanup, but it shouldn’t be too bad.

Here are the some additions I’d like to make before the midterm if time allows — nothing to be inferred by the ordering except where noted:

1) Support locally defined parameter kind-types (easy), e.g.:


subroutine foo(a,b)
    integer :: DP
    parameter  (DP = 8 )
    integer, parameter :: SP = 4
    real(kind=DP), intent(inout) :: a
    real(kind=SP), intent(in) :: b
    ! ...
end subroutine

2) Next, support use’d module kind-type parameters (easy):


module ktps
  integer :: KTP
  parameter (KTP = 10)
end module ktps
integer function bar(a)
  use ktps
  real(kind=KTP), intent(inout) :: a
  ! ...
end function

3) Next, support module-scoped functions (some corner cases to consider, a bit harder) — just to give you an idea…


module container
  use ktps, local_kpts => KTPS
  integer, parameter :: C_KTPS = local_ktps
  contains
  subroutine mod_subr(a)
    use othermod, rename_p => orig_p
    use yetanother, only: yetanother_p => xxx
    implicit none
    integer(kind=C_KTPS) :: a
    integer(kind=other_mod_param), intent(out) :: b
    integer(kind=rename_p) :: c
    ! ...
  end subroutine
end module

4) Logical <-> Integer conversions (probably before the above)

The ‘logical’ type is basically a specially typed integer — whenever a subprogram’s argument is a logical, its C wrapper should use the equivalently-sized integer and do a conversion before & after the call.

5) Character arguments:

It seems that these are functionally equivalent to one-dimensional arrays, with a few bugaboos to watch out for. I’m lumping them in with the second-half of the project, since after working with integer/real arrays I’ll have a better feel as to the best way to handle character arrays.

6) Complex arguments:

Since we can’t assume C99 _Complex support, these need to be passed in as structs, converted to the native complex type in fortran, and converted back on return. Once (4) is done, I’ll have a better feel for these. I’m putting them off until the second half, since they’re equivalent to passing derived types.

Cython side:

I’m turning my full attention to tickets 299, 300 & 178 now, provided that the recent intense discussion about a native Cython array type leave these tickets unchanged (see http://article.gmane.org/gmane.comp.python.cython.devel/6654).

Wojciech Walczak: gminick

Thu, 18 Jun 2009 18:00:13 +0000

OK, it is now possible to login with one’s OpenID login after the docs and webapp for serving the documentation were generated. It took me some time to make it work, mainly because Paste’s module for OpenID logging – paste.auth.open_id – is rather out of date.

What I needed to do to make it work was to put together the WSGI architecture and the OpenID logging logic. The examples from python-openid package were very helpful, though. It is working already, but there are some problems I still need to resolve. To name a few: add a way to distinguish admins from ordinary users (anyone can log in, but only choosen users have special rights), add a “logged in as…” line to the templates (the problem here is that the docs are used as templates twice: when Sphinx generates the docs and later on when webapp is displaying them).

Nevertheless, you can try it just as described in earlier posts. Also, until June 22 a form for posting comments should be ready. Seems like busy weekend!

James Pruitt

noreply@blogger.com (Eric) — Thu, 18 Jun 2009 15:30:00 +0000

After contacting Josiah Carlson, author of the activestate patch for asynchronous I/O in subprocess.Popen, he was more than happy to release the code into the public domain so my license conflicts with his patch are now resolved. Some of the tests I wrote for his patch have failed and he agreed to take a look at what I wrote to help me get the tests running properly.

Upon running my tests, he realized that pywin32 is not part of the standard library which complicates my project a bit as far as Windows is concerned so I am now looking for alternatives to Josiah Carlson's asynchronous I/O patch for subprocess.Popen on Windows. If I cannot find an alternative, I will be using cTypes with the parts of Mark Hammond's code that I need, license permitting. Any suggestions are greatly appreciated.

Dale Peterson: NSF Success!

Wed, 17 Jun 2009 19:19:42 +0000

Today I found out that our NSF proposal to the control systems division was accepted! This means I will be funded for two years to study and work bicycles and control! This is a dream come true for me and I am very excited about it.

Soon, I hope to use PyDy to derive the model of the bicycle for our studies. Once trigsimp() is made better, it will be fully capable of this

James Pruitt: My Original Proposal

noreply@blogger.com (Eric) — Wed, 17 Jun 2009 15:50:00 +0000

My proposal was never posted so I am putting it up now.

Proposal

* A long running issue with the subprocess module is the handling of asynchronous io (http://bugs.python.org/issue1191964, http://ivory.idyll.org/blog/feb-07/problems-with-subprocess). Several patches have been proposed and several alternative solutions suggested. The patches and the alternative solutions will be implemented initially for 2.7 and then for 3.0. The subprocess test suite will be augmented to fully test this new functionality.

* Reimplement the commands module in terms of subprocess functions (http://ivory.idyll.org/blog/mar-07/replacing-commands-with-subprocess). Since the commands module will soon deprecated, this will provide a platform independent transition to those still using the commands module.

The modifications to the subprocess module will be presented initially for code review on Rietveld. A message will be added to the above referenced issues so that all interested parties can comment.

Schedule

Start of program:

* Implement command modules using subprocess
* Extend test suite
* Post code review

Midterm evaluation:

* Address issues from code review and post patch
* Synthesize the patches from the above links
* Extend test suite
* Post code review

Final evaluation:

* Address issues from code review and post patch

Kang Zhang: Keyring Lib on OSX

Jobo.ZH@gmail.com (Kang) — Wed, 17 Jun 2009 11:05:00 +0000

Python Keyring Lib has been created on bitbucket.org. Anyone who has interest in this project can check out the latest code from that address using Mercurial.

We've finished the basic part on Mac OS X. It supports getting and setting password in Keychain.

Following screenshots show a demo. The demo stored the password, and read it from the keychain. Notice that OS X prompted a dialog to let the user unlock the keychain.

The source code of this demo can be accessed from test/demo.py in the repository.

Priit Laes: GSoC Week 2/3

Tue, 16 Jun 2009 11:33:51 +0000

As I have still exam session in progress, I haven't had much motivation to write about my almost non-existent progress with GSoC.

So, school first: I have still one exam left - Optics, which is on 19th and will be hard as some of the stuff we are required to know does not even exist (tunnel effect) in the Hecht's "Optics". (Although I might be wrong, as I only have 2nd edition and later editions might have updates). Fortunately I have almost 100% of the lecture notes available. After the exam, either on Friday or Saturday, I will be moving back home (which is ~1 day of travelling) and then I can really start with GSoC.

Now, GSoC..

As some people have had trouble connecting to my personal server (Who said that free countries do not have internet censorship?), I created a mirror of my private Sympy repo on github. There you can find the following two branches I consider upstream quality:

py3k - some preliminary Python3K porting, which gets rid of all the dict.has_key() that were reported during test phase. (So some still remain).
doc-fixes - I ran into few problems while tried to build Sympy's documentation - most of them were Sphinx warnings, although ran into some errors too. So please review and comment.

There's also my pde-wip branch, but currently I'm not very happy with that approach...

Thanks Ondrej for pushing me to keep posting :)

Dale Peterson: sin, cos, tan, cot, sec, csc

Tue, 16 Jun 2009 02:29:02 +0000

In my trigsimp branch of Sympy, I’ve modified all the .eval methods of sin(), cos(), tan(), and cot() so that they give the same behavior of Mathematica. Next on the list is to implement sec() and csc() in a similar fashion. Currently sec() and csc() do not exist in Sympy, so I will have to familiarize myself with all the other code that is in the trig functions that do exist. It seems to me that much of the code is redundant and may not be tested for, so I will attempt to do some housecleaning in that section while I am at it. The rule based trig simplification should then work fine, I have already started some basic tests of applying the rules that Fu et al. show in their examples, and Sympy handles them just fine.

Ondrej also helped me get trigsimp branch up and running on github:
git://github.com/hazelnusse/sympy.git

My trigsimp branch is there and I have created an issue for it here:
http://code.google.com/p/sympy/issues/detail?id=1475

M. Shuaib Khan: Gcov and Python3k C coverage

noreply@blogger.com (M. Shuaib Khan) — Tue, 16 Jun 2009 02:19:52 +0000

Gcov is a very handy tool. I've been using it for a while now to manually trace the C code coverage of different Python 2.6 modules.

Gcov Usage:

To use gcov with a C source file for code coverage, you have to compile your code with GCC and pass it two arguments at the compile time, "-fprofile-arcs -ftest-coverage". For example if you have a source file named "main.c", you would compile it as:

$gcc -o main -fprofile-arcs -ftest-coverage main.c

The compiler will compile your source file and produce another file named "main.gcno". Now run:

$gcov main

You will see that gcov will report 0% code coverage as you haven't run the compiled code yet. Run the compiled program and then use gcov for code coverage report again:

$./main
$gcov main

Depending on how much code in your program is executed, gcov will report the code coverage percentage, and will also produce two new files named "main.c.gcov" and "main.gcda". "main.c.gcov" is the file that holds the record of what lines were executed in your source file and how many times.

Using Gcov with Python:

There is a bit of a problem with Gcov. You need to statically link all of your code into your program in order to get code coverage analysis on it. Python doesn't do that automatically, and there is no easy configure switch to do it. So you have to manually play with the configurations to get different modules statically linked in. To compile a module statically, one way to do is to copy its entry from Modules/Setup into Modules/Setup.local, and put it under a "*static*" heading (without the quotes).

Lets say you want to compile the "mmap" module statically. First just start your python interpreter and import mmap. Now check for the __file__ attribute of the module:

>>>import mmap
>>>print mmap.__file__

The interpreter will print the module's location on the hard drive from where it was imported. Lets compile this module statically into the python interpreter now. We copy its entry from the Modules/Setup file in the python source tree into Modules/Setup.local, and put it under a *static* heading:

*static*
mmap mmapmodule.c -I$(prefix)/include -fprofile-arcs -ftest-coverage -L$(exec_prefix)/lib -lz -lgcov

The part after mmapmodule.c is to make this module work with gcov so we could get coverage report on it. Now compile your python interpreter and look for mmap's __file__ attribute again. This time you will see no such attribute defined for the module, which means it is built in this time.

Changing current directory into Modules and listing files will show that gcov has produced its data files for the mmapmodule. Run gcov on it to get the current coverage reports.

$gcov mmapmodule

You might get some code coverage for it since the import of the module into your python environment runs some of the initialization code. Had you run gcov on it without importing it first, you would have got 0% code coverage. Here is what I get:

File '/usr/include/sys/sysmacros.h'
Lines executed:0.00% of 6
/usr/include/sys/sysmacros.h:creating 'sysmacros.h.gcov'

File '/usr/include/sys/stat.h'
Lines executed:0.00% of 12
/usr/include/sys/stat.h:creating 'stat.h.gcov'

File './Modules/mmapmodule.c'
Lines executed:0.00% of 476
./Modules/mmapmodule.c:creating 'mmapmodule.c.gcov'
./Modules/mmapmodule.c:cannot open source file

Now lets run the mmap testing code and see how much of the actual code this test suite exercises:

$./python Lib/test/test_mmap.py -v
$cd Modules
$gcov mmapmodule

File '/usr/include/sys/sysmacros.h'
Lines executed:0.00% of 6
/usr/include/sys/sysmacros.h:creating 'sysmacros.h.gcov'

File '/usr/include/sys/stat.h'
Lines executed:0.00% of 12
/usr/include/sys/stat.h:creating 'stat.h.gcov'

File './Modules/mmapmodule.c'
Lines executed:71.64% of 476
./Modules/mmapmodule.c:creating 'mmapmodule.c.gcov'
./Modules/mmapmodule.c:cannot open source file

71.64% of the module is exercised by the test suite.

What's Next:

I have traced the code coverage of a number of modules by now. Next is to look for a way to easily compile a major number of modules statically into the Python interpreter so to easily get an automated code coverage report on it. This can be done by uncommenting appropriate entries for the modules in Module/Setup file, but I've been getting into a few problems with it right now. Solving that is the first task, and then the part of making figleaf do all of this automatically.

Skipper Seabold: Design Issues: Understanding Python's super

noreply@blogger.com (jseabold) — Mon, 15 Jun 2009 22:34:03 +0000

The current statistical models package is housed in the NiPy, Neuroimaging in Python, project. Right now, it is designed to rely on Python's built-in super to handle class inheritance. This post will dig a little more into the super function and what it means for the design of the project and future extensions. Note that there are plenty of good places to learn about super and that this post is to help me as much as anyone else. [*Edit: With this in mind, I direct you to Things to Know about Python Super if you really want a deeper and correct understanding of super. This post is mainly a heuristic approach that has helped me in understanding basic usage of super.] You can find the documentation for super here. If this is a bit confusing, it will, I hope, become clearer after I demonstrate the usage.

First, let's take a look at how super actually works for the simple case of single inheritance (right now, we are not planning on using multiple inheritance in the project) and an __init__ chain (note that super can call any of its parent class's methods, but using __init__ is my current use case).

The following examples were adapted from some code provided by mentors (thank you!).

class A(object):
def __init__(self, a):
self.a = a
print 'executing A().__init__'

class B(A):
def __init__(self, a):
self.ab = a*2
print 'executing B().__init__'
super(B,self).__init__(a)

class C(B):
def __init__(self, a):
self.ac = a*3
print 'executing C().__init__'
super(C,self).__init__(a)

Now let's have a look at creating an instance of C.

In [2]: cinst = C(10)
executing C().__init__
executing B().__init__
executing A().__init__

In [3]: vars(cinst)
Out[3]: {'a': 10, 'ab': 20, 'ac': 30}

That seems simple enough. Creating an instance of C with a = 10 will also give C the attributes of B(10) and A(10). This means our one instance of C has three attributes: cinst.ac, cinst.ab, cinst.a. The latter two were created by its parent classes (or superclasses) __init__ method. Note that A is also a new-style class. It subclasses the 'object' type.

The actual calls to super pass the generic class 'C' and a handle to that class 'self', which is 'cinst' in our case. Super returns the literal parent of the class instance C since we passed it 'self'. It should be noted that A and B were created when we initialized cinst and are, therefore, 'bound' class objects (bound to cinst in memory through the actual instance of class C) and not referring to the class A and class B instructions defined at the interpreter (assuming you are typing along at the interpreter).

Okay now let's define a few more classes to look briefly at multiple inheritance.

class D(A):
def __init__(self, a):
self.ad = a*4
print 'executing D().__init__'
# if super is commented out then __init__ chain ends
#super(D,self).__init__(a)

class E(D):
def __init__(self, a):
self.ae = a*5
print 'executing E().__init__'
super(E,self).__init__(a)

Note that the call to super in D is commented out. This breaks the __init__ chain.

In [4]: einst = E(10)
executing E().__init__
executing D().__init__

In [5]: vars(einst)
Out[5]: {'ad': 40, 'ae': 50}

If we uncomment the super in D, we get as we would expect

In [6]: einst = E(10)
executing E().__init__
executing D().__init__
executing A().__init__

In [7]: vars(einst)
Out[7]: {'a': 10, 'ad': 40, 'ae': 50}

Ok that's pretty straightforward. In this way super is used to pass off something to its parent class. For instance, say we have a little more realistic example and the instance of C takes some timeseries data that exhibits serial correlation. Then we can have C correct for the covariance structure of the data and "pass it up" to B where B can then perform OLS on our data now that it meets the assumptions of OLS. Further B can pass this data to A and return some descriptive statistics for our data. But remember these are 'bound' class objects, so they're all attributes to our original instance of C. Neat huh? Okay, now let's look at a pretty simple example of multiple inheritance.

class F(C,E):
def __init__(self, a):
self.af = a*6
print 'executing F().__init__'
super(F,self).__init__(a)

For this example we are using the class of D, that has super commented out.

In [8]: finst = F(10)
executing F().__init__
executing C().__init__
executing B().__init__
executing E().__init__
executing D().__init__

In [8]: vars(finst)
Out[8]: {'ab': 20, 'ac': 30, 'ad': 40, 'ae': 50, 'af': 60}

The first time I saw this gave me pause. Why isn't there an finst.a? I was expecting the MRO to go F -> C -> B -> A -> E -> D -> A. Let's take a closer look. The F class has multiple inheritance. It inherits from both C and E. We can see F's method resolution order by doing

In [9]: F.__mro__
Out[9]:
(<class '__main__.F'>,
<class '__main__.C'>,
<class '__main__.B'>,
<class '__main__.E'>,
<class '__main__.D'>,
<class '__main__.A'>,
<type 'object'>)

Okay, so we can see that for F A is a subclass of D but not B. But why?

In [10]: A.__subclasses__()
Out[10]: [<class '__main__.B'>, <class '__main__.D'>]

The reason is that A does not have a call to super, so the chain doesn't exist here. When you instantiate F, the hierarchy goes F -> C -> B -> E -> D -> A. The reason that it goes from B -> E is because A does not have a call to super, so it can't pass anything to E (It couldn't pass anything to E because the object.__init__ doesn't take a parameter "a" and because you cannot have a MRO F -> C -> B -> A -> E -> D -> A as this is inconsistent and will give an error!), so A does not cause a problem and the chain ends after D (remember that D's super is commented out, but if it were not then there would be finst.a = 10 as expected). Whew.

I'm sure you're thinking "Oh that's (relatively) easy. I'm ready to go crazy with super." But there are a number of things must keep in mind when using super, which makes it necessary for the users of super to proceed carefully.

1. super() only works with new-style classes. You can read more about classic/old-style vs new-style classes here. From there you can click through or just go here for more information on new-style classes. Therefore, you must know that the base classes are new-style. This isn't a problem for our project right now, because I have access to all of the base classes.

2. Subclasses must use super if their superclasses do. This is why the user of super must be well-documented. If we have to classes A and B that both use super and a class C that inherits from them, but does not know about super then we will have a problem. Consider the slightly different case

class A(object):
def __init__(self):
print "executing A().__init__"
super(A, self).__init__()

class B(object):
def __init__(self):
print "executing B().__init__"
super(B, self).__init__()

class C(A,B):
def __init__(self):
print "executing C().__init__"
A.__init__(self)
B.__init__(self)
# super(C, self).__init__()

Say class C was defined by someone who couldn't see class A and B, then they wouldn't know about super. Now if we do

In [11]: C.__mro__
Out[11]:
(<class '__main__.C'>,
<class '__main__.A'>,
<class '__main__.B'>,
<type 'object'>)

In [12]: c = C()
executing C().__init__
executing A().__init__
executing B().__init__
executing B().__init__

B got called twice, but by now this should be expected. A's super calls __init__ on the next object in the MRO which is B (it works this time unlike above because there is no parameter passed with __init__), then C explicitly calls B again.

If we uncomment super and comment out the calls to the parent __init__ methods in C then this works as expected.

3. Superclasses probably should use super if their subclasses do.

We saw this earlier with class D's super call commented out. Note also that A does not have a call to super. The last class in the MRO does not need super *if* there is only one such class at the end.

4. Classes must have the exact same call signature.

This should be obvious but is important for people to be able to subclass. It is possible however for subclasses to add additional arguments so *args and **kwargs should be probably always be included in the methods that are accessible to subclasses.

5. Because of these last three points, the use of super must be explicitly documented, as it has become a part of the interface to our classes.

Skipper Seabold: Legendre on Least Squares

noreply@blogger.com (jseabold) — Mon, 15 Jun 2009 22:30:46 +0000

I found the epigraph to Åke Björk's Numerical Methods for Least Squares Problems to be a nice intersection of my interests.

De tous les principes qu'on peut proposer pour cet object, je pense qu'il n'en est pas de plus general, de plus exact, ni d'une application plus facile que celui qui consiste à rendre minimum la somme de carrés des erreurs.

Of all the principles that can be proposed, I think there is none more general, more exact, or of an easier application than that which consists of rendering the sum of squared errors a minimum.

Adrien Marie Legendre, Nouvelles méthodes pour la détermination des orbites des comètes. Appendice. Paris, 1805.

Danilo Freitas: Progress

dsurviver@gmail.com (Danilo Freitas) — Mon, 15 Jun 2009 12:08:01 +0000

Some features were already implemented on Cython.
Until now, we already have:
- C++ class declaration
- C++ class attributes declaration
- Multiple inheritance
- Declaration of inherited attributes
- Use of namespace
The next step now is allowing attribution of inherited objects.

Fabian Pedregosa: The boolean satisfiability problem

Mon, 15 Jun 2009 04:00:20 +0000

Most annoying problem in my implementation of the query system is that it will not solve implications if the implicates are far away from each other. For instance, if the graph of known facts is something like this

 Integer ----> Rational --> Real --> Complex
   ^  ^
   |  |
   |   -------
   |         |
 Prime      Even
   ^
   |
   |
 MersennePrime

Then it will not know how to handle the query: Is x complex assuming it is a Mersenne prime ?. This is because the vertices MersennePrime and Complex are far away from each other and the query function does not load the complete graph of known facts, but rather a small subgraph centered on the assumed facts …

This was done so for efficiency reasons, because in the initial implementation I feared that the graph of known facts could become huge and thus making it unfeasible to search into.

But things have changed now. Known facts is not huge at all, roughly having over 20 vertices, so it is feasible to build the complete graph the first time query() is called and store it for future uses. And, most important, we have implemented fast algorithms for the problem of boolean satisfiability (DPLL under sympy.logic.algorithms.dpll), so all is ready to implement these ideas in the following days.

Interestingly, there seems to me many open source libraries for solving this problem. One that caught my attention early is MiniSAT, a nice little program written in C++ which is really fast

Andrew Friedley

noreply@blogger.com (Andrew Friedley) — Sun, 14 Jun 2009 11:16:37 +0000

Have been busy lately.. big paper deadline (CorePy paper!) and moving. When I can, I've been working on general code correctness/cleanliness/robustness. Management and movement of various data inside the 'corefunc' C module has been cleaned up. I now have a fairly generic routine for creating a ufunc from some type information and CorePy InstructionStreams. Multi-core is supported at the C level in a generic way; nothing there is ufunc-specific.

I didn't mention this before, but my code previously only worked correctly for input sizes that were a multiple of the number of threads used (1, 2, 4) and the number of iterations that were unrolled (1, 4, 8). Now my code works for any size input. At the thread level, I just had the last thread (if more than one) check if the number of threads evenly divided the input length. If not, that last thread increased the number of elements it processed to compensate. At the CorePy loop level, a special code path was needed to handle both the case where the input length is less than the number of unrolled loop iterations, and when the length was not a multiple of the unrolled iterations. Here, I do some fancy (but somewhat messy) checks, and added unrolled code to handle one element at a time (as opposed to 4/8 at a time using SIMD).

Turns out reduction wasn't working right when multiple threads are used, but this is fixed now (took a while! evil bugs). Each thread reduces (eg. adds, subtracts) all the elements in its assigned input sub-range into a local register. When complete, the local value is atomically reduced into the single shared output value residing in memory. CorePy's support for the LOCK prefix (makes some instructions atomic) wasn't complete, so I had to go do that. I think I've covered most/all of the instructions supporting the LOCK prefix on x86_64.

This brings up sort of a side issue -- because floating point is not associative, the use of multiple threads accumulating to their own local values independently means the order of operations for reductions is changed. This causes a result that is different from NumPy's. I'm not sure what to do about this, if anything -- I don't think there is a way to maintain thread-level parallelism and perform the operations in the same order as NumPy. I think all I can do is provide a warning about this issue when using multiple threads to do reductions.

Sort of related to this GSOC work, I implemented ExtBuffer, an object implementing the Python 2.6 buffer interface (mainly for use with NumPy arrays). ExtBuffer provides guaranteed page-aligned memory, supports huge pages, and can optionally just take a pointer to some pre-allocated memory and use that (useful for overlaying special memory-mapped hardware regions, e.g. Cell SPU local store). Documentation here:

http://corepy.org/wiki/index.php?title=Extended_Array

I was actually anticipating that I would have to write ExtBuffer as part of this GSOC project. The issue was that SIMD code typically requires 16-byte memory alignment for best performance. To my knowledge, malloc() (which I understand NumPy arrays use to alloc memory) only guarantees 8-byte alignment. However I've just been using NumPy arrays as-is and have not (yet) run into any alignment issues. So I need to look into this and see if I can now really rely on NumPy array data always being at least 16-byte aligned.

Where to next?

I really just have an addition ufunc with support for in64 and float32 types right now. This has sufficed for building the initial codebase, but now I need to expand this to support more data types and more operations. With that done, it'll be time to start refactoring duplicate code and designing/building more of a framework to build new ufuncs quickly/easily (my halfway milestone!)

Tyler Laing: Champions Online Contest Entry

Sun, 14 Jun 2009 04:46:35 +0000

So apparently Tim Absath of Ctrl-alt-delete comic is running a contest for five beta keys for Champions Online, being produced by the excellent Cryptic Studios. The contest requires inventing a superhero, and here’s my entry: Once a support technician for a tech startup, Max Kellder is now unrecognizeable. Following the clues of a new mysterious, and [...]

Aaron Meurer: asmeurer

Sat, 13 Jun 2009 00:45:44 +0000

I am on vacation now. I am visiting the Grand Canyon, Hoover Dam, Yellowstone, and other places. I will be able to do some coding on the road, but not as much as when I am home. The vacation will last about a week. Hopefully I will be able to get a blog post about what I have been doing in SymPy written up in the next few days.

Tyler Laing: Movie Module Progress: Week 4

Fri, 12 Jun 2009 21:25:19 +0000

This week was a bit of varied work. I did this week: Cleaned up, refactored, and commented code. This took a day Hunting down a memory leak. This took two days. It ended up being leaving opening a video stream without closing it before looping. Worked on getting loops to work properly. This took the better part of a [...]

Freddie Witherden: Project Update

noreply@blogger.com (Freddie Witherden) — Fri, 12 Jun 2009 20:14:53 +0000

Sorry for the lack of updates over the last couple of weeks — however I have had my first year undergraduate exams this week and was revising the week prior. With my exams now out of the way I am able to start working full-throttle on my GSoC project.

According to my schedule (which I will post verbatim in a later post) the plan it to spend the first week getting up to speed on exactly how the TeX layout algorithm works and how it is currently implemented in Matplotlib. If all goes to plan you can expect a diagram-filled blog post on it near the end of next week.

My mentors, John Hunter and Michael Droettboom have given me some good reading material on it (along with the current implementation, of course) so this should not be too difficult.

Finally, sometime in the near future you can also expect a discussion on backends (so the code responsible for turning a list of characters and coordinates into an image/document).

James Pruitt: Who will police the police?

noreply@blogger.com (Eric) — Fri, 12 Jun 2009 17:18:00 +0000

It appears the recent release of Python 3.0 have a buggy pdb. When I first ran into the issue, I thought it might have had something to do with my unit tests or possible incompatibilities between the subprocess.Popen module and pdb. I wrestled with it before finally throwing a bit of "Google-Fu" at it and discovered that I was not the only person having this issue. (http://bugs.python.org/issue6126)

Python 2.7 is supposed to be a 2.X compatible backport of the features and syntax found in Python 3.0 so any code added to the 2.7 branch must be either a bug fix or a backport of a Python 3.0 feature. Before I learned about that, my proposal was directed towards Python 2.7. The pdb module in Python 2.7 works flawlessly and did not give me any trouble while I was debugging my code with it.

Until the issues with the Python 3.0 pdb module are resolved, I will be taking advantage of the forward compatiblity and coding in Python 2.7 so that I will have a functional code debugger.

Fabian Pedregosa: Initial implementation of the query system

Fri, 12 Jun 2009 04:36:23 +0000

I sent some patches to sympy-patches with an initial implementation of the query system.

You can check it out by pulling from my branch:

git pull http://fseoane.net/git/sympy.git master

into your sympy repo.

Some examples of what you can do (sample isympy session):

In [1]: query(x, positive=True)

Returns None, as we do not know whether x is positive or not.

In [2]: query(abs(x), positive=True) Out[2]: True

because abs() is always positive. Because exp() is always positive, the following should also be True:

In [3]: query(exp(x), positive=True)

but why then does it return None ?. Well, it simply is not True that exp() is always positive, it is always positive for real values, but SymPy does not assume that x is real, so you would have to specify that. This is now done with the keyword assumptions:

In [5]: query(exp(x), positive=True, assumptions=Assume(x, real=True)) Out[5]: True

As you can see, now assumptions are independent objects and are not tied to symbols any more. For more examples, see the file sympy/query/tests/test_query.py

still in the TODO list:
- support for global assumptions
- solve more complex implications, like query(x, positive=True, assumptions=Assume(x, even=True)) where it should build the chain of implications even => integer => rational => real. This chain of implications currently stops at rational for efficiency reasons, because the number of facts grows in each step which makes the number of possible paths grow exponentially.

Wojciech Walczak: gminick

Thu, 11 Jun 2009 14:46:32 +0000

It already works. I still can point to some problems, but the Xapian search engine is quite usable and it acknowledges themes choosen by the user.

A “search” form for Xapian is placed under a traditional “quick search” form. You can already build your documentation with this addition. Simple “make webapp” makes the job.

Kurt Smith: freemalloc

Thu, 11 Jun 2009 01:46:33 +0000

Much progress over the past few weeks, I’m glad to say. Here are the main threads that are out there.

First, the 10000 meter view: For my GSoC, I’m developing, under the guidance of Dag Sverre Seljebotn, (1) a python package that wraps fortran so it can be easily called from a cython program; (2) improving cython’s buffer support to allow the passing of buffers/arrays to external code (C, fortran, or whatever can be linked in) or cdef functions.

The fortran-side:

Dag Sverre and I discussed a bit about the naming of the fortran wrapper project — right now it’s under the working title “f2cy,” in the long tradition of “<some-language/protocol/format>2<other-language/protocol/format>” converter names. There are at least 85 similarly-named executables on my ubuntu system (apropos 2 | awk ‘{ print $1 }’ | grep ‘[a-zA-Z]2[a-zA-Z]‘ | wc -l). Since the wrapper and the code it generates has very little direct dependence on python/cython, and since it may be useful for those who want to automate wrapping fortran inside a C program, the name will likely change. I’m thinking of something along the lines of ‘fwrap’ — Dag suggested ‘fbridge’ to emphasize the connectivity aspect.

f2cy, which is using the fortran parser ‘fparser’ from the latest version of f2py (http://code.google.com/p/f2py/) is able to parse pretty much everything required for the midterm review. The rest of f2cy is still rapidly developing. The package lives inside the Tools directory of the cython distribution package. To take a look (it’s still in pre-alpha stage) checkout the mercurial repo here: http://hg.cython.org/gsoc-kurt/.

After fparser generates the parse tree, f2cy makes a few passes on the AST (using the Visitor pattern); one pass adds type ‘annotations’ to the dummy arguments of the subprograms to be wrapped, the next pass generates the autoconfig fortran source file, the next one generates the fortran wrapper. Only scalar-valued arguments are supported for now, but by August f2cy will support arrays (assumed-shape, assumed-size and explicitly shaped) and passing derived types (structs) end-to-end from cython/C. How is this done? Thanks to the iso_c_binding intrinsic module within the Fortran 2003 standard, a Fortran program can be portably linked to C code without too much pain — certainly less pain than was required before. This module allows f2cy to easily wrap subprograms that take scalars, arrays and derived types — in the distant future, f2cy may support passing C callbacks. There still is an issue of resolving the kind-type-parameters for the dummy arguments, but that is what the autoconfig file is for.

Our goal in the project is to generate fortran wrappers that are completely portable, in the same way that Cython itself is portable. A number of projects will generate an extension module with Cython and ship it without a Cython dependence, so the source has to work with any C compiler. In the same way, f2py will generate wrappers that do not have any compiler-specific knowledge (including the particular values for the kind-type-parameters for the various types). This isn’t purely academic, since some compilers define integer type KTPs consecutively (1,2,3,…) while for others (gfortran, xlf) the KTPs correspond to the byte-size (1,4,8,10,…). This makes for some complications and makes the autoconfig file necessary. More on the autoconfig in a future post.

I’ll address the cython-side in a short while.

Jeff Ling: Draw SVG Images In Python

Wed, 10 Jun 2009 08:37:28 +0000

Draw SVG Images In Python « ActiveState Code.

Right now I’m working on making my widget draw to an svg image for the later SXE implementation. It’s not the funnest part of this project but it isn’t too bad.

Danilo Freitas: Internet problems :(

dsurviver@gmail.com (Danilo Freitas) — Tue, 09 Jun 2009 13:41:32 +0000

Since Friday, May 29, I'm without internet at home. I'm using internet on university these days. I hope it come back soon.
It's a really bad I have this problem right now. It's obviously not helping on nothing.