Mono is an open source version of the .NET framework; it allows you to run .NET applications not just on Windows but on Linux and the Mac. I've spent quite some time over the last week getting our Python spreadsheet, Resolver One, to run on the Windows version, and thought it would be worth sharing some experiences.
Some background first: one of our long-term goals at Resolver Systems is to get our currently Windows-only products working on other platforms. Everything's coded in .NET, so in an ideal world we'd just be able to run it all under Mono. However, there are two problems:
- Some of the third-party components we use are ".NET" in the sense that they offer us a .NET interface, but under the hood they call down to lower-level Windows functions using P/Invoke. Because they're using Windows-specific stuff, they won't run on non-Windows operating systems, even with Mono.
- As always with these things, while there is a formal specification for what .NET implementations like Microsoft's .NET or Mono are meant to do (called the CLI), implementations differ due to bugs, things that are awaiting implementation, or ambiguities in the spec.
Obviously, we need to handle both of these kinds of problem to successfully port our software. We're handling the first kind by moving over to newer, "pure" .NET components, for example by writing our own grid. But we didn't want to finish all that work and only then discover all of the problems caused by the second kind of incompatibility. Now, Mono does support P/Invoke, so while the first kind of problem clearly prevents us from running on Mono right now on, say, a Mac, it does not prevent us from running on Mono on Windows. So we decided to do that simpler port in parallel with our development of the new components, so that we could find out any nasty issues of the second kind as early as possible.
First things first: it really was surprisingly easy! All kudos to the Mono developers, this really is an example of an open source project that works really well. The problems below are really low-level details, and most of them are unlikely to hit anyone apart from us. However, it's worth enumerating them, at least for posterity's sake — and perhaps they'll be helpful for people Googling for solutions to similar obscure problems.
Problem 1: A change to the process binary
The first problem we hit was our code to load up our DLLs. Resolver One is comprised of quite a few libraries, and we need to be careful to load the specific ones that it's shipped with rather than others that might be on the user's path. We do this by finding our binary's location, using
Path.GetDirectoryName(Process.GetCurrentProcess().MainModule.Filename), and then using
Assembly.LoadFile(filename) to load the DLLs explicitly rather than the more normal
clr.AddReference, which uses the path.
The problem is, when you run a .NET application under Mono, the current process's binary is not (say)
Resolver-One.exe, but instead
mono.exe. So Resolver One was trying to find its libraries in the Mono install directory, which obviously didn't work. In the short term, we were able to work around this by hard-wiring the DLL path. In the longer term, we'll have to do something more clever...
Problem 2: ImeModeBase
Once we'd fixed the first problem we got a new error:
Could not find "System.Windows.Forms.Control.get_ImeModeBase". A bit of investigation made it clear that the current version of Mono doesn't support this method (though when it does, the target of that link will show it). It looks like the method was introduced in the (relatively recent) .NET 2.0 SP2, and presumably it will be implemented sometime, but it's not there right now.
The question was, could Resolver One run without this method? The answer seemed likely to be "yes", as we were able to run on earlier versions of .NET 2.0. We took a look at our codebase and tried to find out what it was that was referencing the method. It turned out to be our "precompiled subclass" DLLs. These are something we introduced a while back to improve our startup performance; simplifying a bit, what happens is that when we package up Resolver One for distribution, we run a compile process on all of the classes in our code that are subclasses of .NET components. Doing this process once before we release the software means that it's not done every time Resolver One starts up, with obvious performance benefits. The downside is that the compiled subclasses have explicit references to the members of their .NET superclasses, whether they use them or not. And because we run the compilation process under .NET 2.0 SP2, our compiled subclasses wind up with explicit references to stuff that doesn't exist in earlier versions of .NET, or (as it turns out) in Mono.
The good news is that if you're willing to take a performance hit, the subclass compilation can be made optional. This isn't something we put in the production version of Resolver One right now, as it's Windows only (and so there's little point in having a "start up more slowly for no useful reason" command-line option), but it was easy to add in. Using non-precompiled subclasses got us past this problem. Perhaps our future Linux/Mac-compatible version can use subclasses that are precompiled against Mono.
Problem 3: System.Reflection:MonoGenericMethod
This one was the easiest one to find out how to work around, but took the longest time. Once we'd got past the precompiled assembly problem, trying to start Resolver One gave us a dialog saying "** ERROR **: Can't handle methods of type System.Reflection:MonoGenericMethod aborting...". Mono then bombed out with a Windows "application has stopped working" dialog.
A bit of Googling followed, and we were delighted to discover that a bug that triggered this exact error message had been fixed just ten days previously on the Mono trunk and the 2.6 branch. There's luck for you.
Unfortunately we also discovered that the Mono team don't release nightly builds of their Windows installer, so the only ways to get this fix would be either to wait until the 2.6.5 release, or to build it ourselves. Being impatient by nature, we decided to do the latter, and this took quite a while. I'll post separately about what we had to do, as it may be useful for others; there's a lot of excellent documentation on building Mono for Windows, but some of it's a bit out of date. Luckily, the people on the Mono email list are friendly and helpful, so we were able to get there in the end.
So, after a bit of work we had a working version of Mono built from the tip of the 2.6 branch.
Problem 4: Logfile locations
When we started up Resolver One with the new Mono, we got the error
SystemError: Access to the path "C:\ResolverOne.log" is denied. This was interesting, because our default logfile location is determined using
Path.GetTempPath(). I'm not sure where Mono picks up the value for that, but presumably it was returning an empty string. Perhaps we were missing something in our environment variables? Either way, we decided to work around it by using Resolver One's
--logfile command-line option.
Problem 5: Madness in our methods
When told to log to an appropriate directory, Resolver One started up, and things started looking pretty good! The splash screen appeared, the "starting" progress bar moved and then... it crashed. The log file had this in it:
Exception: Method DLRCachedCode:FormulaLanguage.parsetab$16 (IronPython.Runtime.CodeContext,IronPython.Runtime.FunctionCode) is too complex. CLS Exception: System.InvalidProgramException: Method DLRCachedCode:FormulaLanguage.parsetab$16 (IronPython.Runtime.CodeContext,IronPython.Runtime.FunctionCode) is too complex. at IronPython.Compiler.OnDiskScriptCode.Run () [0x00000] in
A bit of Googling lead us to two pages that suggested that "too complex" means that the method in question was too long. The module
FormulaLanguage.parsetab is, as you might expect, related to the code we use to parse the formula language — that is, the language in which you write formulae in cells. This language is specified in our code in BNF with associated handler code, and compiled down into Python by the excellent PLY. The parsetab module is the generated code, and as you might expect it has some pretty unreadable stuff in it; there's one dictionary literal that is on one 81,000-character line.
The easy fix to work around this problem was to split
parsetab.py up into multiple modules. There were three variables that were being initialised with oversized literal expressions,
_lt_productions. We created a separate module for each, which simply contained the initialisation code for the specific variables:
lt_productions_file.py. Finally, we replaced the code in
parsetab.py that had been moved to the new files with appropriate import statements: for example, from
lt_action_items_file import _lt_action_items.
This fixed the problem, and allowed us to move onto the next one! It's not obvious how we'll fix this in the production release, though — the file is auto-generated, so either we'll have to patch PLY or post-process it. Something for us to think about.
Problem 6: Extra
The error we got after fixing the parsetab problem was a bit obscure:
Exception: Finalize CLS Exception: System.Collections.Generic.KeyNotFoundException: Finalize at IronPython.Runtime.PythonDictionary.GetItem (object)The actual location of the error took quite a long time to track down, due to the vagaries of the way we import modules and its effects on stack traces. We eventually wound up having to do a binary chop with log statements in our startup code until we managed to narrow it down to a single line!
It turned out that we have some code that needed to create wrapper functions around all of the functions in a different module. It did this by looping over the values in the dictionary returned by
vars(other_module). However, it didn't want to wrap functions that were internal to .NET, so it had a list of function names to exclude; specifically,
Equals. Clearly these were two function names that had been found by experiment to belong to IronPython modules when running under .NET. The error we were getting was ultimately being caused by IronPython under Mono having just one extra function visible on the module:
Finalize. Adding that to the list of exclusions got us past this error, and on to:
Problem 7: Um... that's it!...on to nothing else! Once we'd fixed the Finalize problem Resolver One started up and ran reasonably happily under Mono on Windows. There were glitches, of course; our code editor component, in particular, didn't like being resized. But the software worked well enough to test, which is all we need for now.
There's obviously a lot to be done before we can get Resolver One running on Macs and Linux machines; the creation of our grid component is going well, but takes time, and we need to do something about the code editor. But the good news is that we've identified the incompatibilities between Mono and Microsoft .NET that will hit us beyond the obvious operating system issues, and there's nothing we can't work around, given a bit of ingenuity. It took a while, but at the end of the day, it was surprisingly easy :-)
Recently at Resolver we've been doing a bit of analysis of the way people, parties and topics are mentioned on Twitter and in the traditional media in the run-up to the UK's next national election, on behalf of the New Statesman.
We've been collecting data, including millions of tweets and indexes to newspaper articles, in a MySQL database, using Django as an ORM-mapping tool -- sometime in the future I'll describe the system in a little more depth. However, from our perspective the most interesting thing about it is how we're doing the analysis -- in, of course, Resolver One.
Here's one little trick I've picked up; using regular expressions in column-level formulae as a way of parsing the output of MySQL queries.
Let's take a simple example. Imagine you have queried the database for the number of tweets per day about the Digital Economy Bill (or Act). It might look like this:
| Date | count(*) | +------------+----------+ | 2010-03-30 | 99 | | 2010-03-31 | 30 | | 2010-04-01 | 19 | | 2010-04-02 | 12 | | 2010-04-03 | 2 | | 2010-04-04 | 13 | | 2010-04-05 | 30 | | 2010-04-06 | 958 | | 2010-04-07 | 1629 | | 2010-04-08 | 1961 | | 2010-04-09 | 4038 | | 2010-04-10 | 2584 | | 2010-04-11 | 1940 | | 2010-04-12 | 3333 | | 2010-04-13 | 2421 | | 2010-04-14 | 1319 | | 2010-04-15 | 1387 | | 2010-04-16 | 3194 | | 2010-04-17 | 860 | | 2010-04-18 | 551 | | 2010-04-19 | 859 | | 2010-04-20 | 685 | | 2010-04-21 | 528 | | 2010-04-22 | 631 | | 2010-04-23 | 591 | | 2010-04-24 | 320 | | 2010-04-25 | 363 | | 2010-04-26 | 232 | +------------+----------+
Now, imagine you want to get these numbers into Resolver One, and because it's a one-off job, you don't want to go to all the hassle of getting an ODBC connection working all the way to the DB server. So, first step: copy from your PuTTY window, and second step, paste it into Resolver One:
Right. Now, the top three rows are obviously useless, so let's get rid of them:
Now we need to pick apart things like
| 2010-03-30 | 99 | and turn them into separate columns. The first step is to import the Python regular expression library:
...and the next, to use it in a column-level formula in column B:
Now that we've parsed the data, we can use it in further column-level formulae to get the dates:
...and the numbers:
Finally, let's pick out the top 5 dates for tweets on this subject; we create a list
...sort it by the number of tweets in each day...
...reverse it to get the ones with the largest numbers of tweets...
...and then use the "Unpack" command (control-shift-enter) to put the first five elements into separate cells.
Now, once we've done this once, it's easy to use for other data; for example, we might want to find the fives days when Nick Clegg was mentioned most on Twitter. We just copy the same kind of numbers from MySQL, paste them into column A, and the list will automatically update:
So, a nice simple technique to create a reusable spreadsheet that parses tabular data.
Today I wrote the code required to call part of the OpenCL API from Resolver One; just one function so far, and all it does is get some information about your hardware setup, but it was great to get it working. There are already .NET bindings for OpenCL, but I felt that it was worthwhile reinventing the wheel -- largely as a way of making sure I understood every spoke, but also because I wanted the simplest possible API, with no extra code to make it more .NETty. It should also work as an example of how you can integrate a C library into a .NET/IronPython application like Resolver One.
I'll be documenting the whole thing when it's a bit more finished, but if you want to try out the work in progress, and are willing to build the interop code, here's how:
- Make sure you have OpenCL installed -- here's the NVIDA OpenCL download page, and here's the OpenCL page for ATI. I've only tested this with NVIDIA so far, so I'm keen to hear of any incompatibilities.
- Clone the dot-net-opencl project from Resolver Systems' GitHub account.
- Load up the
DotNetOpenCL.slnproject file in the root of the project using Visual C# 2008 (here's the free "Express" version if you don't have it already).
- Build the project
- To try it out from IronPython, run
- To try it in Resolver One, load
That should be it! If you want to look at the code, the only important bit is in
DotNetOpenCL.cs -- and it's simply an external method definition... the tricky
bit was in working out which OpenCL function to write an external definition for,
and what that definition should look like.
I've put a slightly tidied version of the notes I kept as I implemented this below, for posterity's sake; if you're interested in finding out how the implementation went, read on...
I'm putting together some spreadsheets that we're going to use to publicise Resolver One over the coming UK electoral campaign, and one set of data I needed was a list of Members of Parliaments' Twitter accounts indexed by an ID that I could use with TheyWorkForYou.com. I was delighted to discover Tweetminster, a site analysing MPs' tweets, and in particular their Twitter list of all UK MPs' accounts, but there was no link to TWFY.
So, given that no-one else seems to have done it, here's my own list:
- A Resolver One file listing Twitter accounts for TheyWorkForYou
- For those of you unlucky enough to not have a copy of Resolver One, here's a CSV file with exactly the same data.
Hope someone finds it useful. It's up-to-date as of this posting, and I'll endeavour to keep it up-to-date, at least for as long as we need it at work :-)
Didrik Pinte has put together a web page on the Python.org Wiki for the London Financial Python Users Group. Only a little content so far, but it will grow... if you're doing financial work in Python in London, do come along to the next meeting -- it will be 7pm next Monday (14 December) at MWB Regent Street, Liberty House 222 Regent Street, London W1B 5TR. You may have to put up with me talking for a while about a spreadsheet you already know everything about, but there will be interesting bits too ;-)
I've done another 3D example in Resolver One. This one uses Yahoo! Finance to download the close prices over the last two years for every stock that's currently in the Dow Jones index, then charts them in a 3D window which you can pan and zoom using the mouse. Here's a video showing it in action (click the image to view):
In my last post about animated 3D graphics in Resolver One (the souped-up spreadsheet the company I work for makes), I showed a bouncing, spinning cube controlled by the numbers in a worksheet. Here's something more sophisticated: a 3D model of the planets in our solar system, also know as an orrery (click the image for video):
I've been playing around with 3D graphics recently, and decided to find out what could be done using .NET from inside Resolver One. (If you haven't heard of Resolver One, it's a spreadsheet made by the company I work for -- think of it as Excel on steroids :-)
I was quite pleased at what I managed with a few hours' work (click the image for video):
Yesterday, my co-founder Robert Smithson presented a fascinating spreadsheet he's built in Resolver One to one of the Financial Times' two UK political correspondents, Jim Pickard. The spreadsheet gives predictions about the next UK general election using a clever methodology Robert has developed, and if you're interested in british politics or clever spreadsheets, you should definitely take a look.
[Update, 27 July] Looks like The Register found the story interesting too!
We kicked off the beta programme for version 1.5 of Resolver One today. It's got some really cool new features, including a console that lets you interact with your spreadsheets from a command-line-style interface, but there's one other change, a tiny one that enables something really interesting -- a combination of the spreadsheet's ease-of-programming with seriously parallel computing that I don't think is really possible with other tools.
We've been in touch with Digipede since Dan Ciruli, their Director of Products, blogged about Resolver One in January 2008. The Digipede Network is a system that allows you to easily code .NET programs that run on a grid of computers -- and he'd set up a Resolver One spreadsheet that was able to call into code running on a Digipede Network to perform part of its calculations, which was particularly impressive given that he only needed to spend five minutes or so putting it together. Looking at what he'd done, I found myself asking "wouldn't it be even cooler if the thing you ran on your compute farm was itself a spreadsheet?"