Crap... this upgrade didn't work out so great. A detailed check of the tables show that the tables are okay. For some reason, the data isn't quite right, and I need to fix up the themes.

Been upgrading from an old Drupal 4.5 (aka Civicspace 82) to Drupal 5.1.

Dropping CiviCRM

I had to get rid of the unused civicrm tables. Installing it seemed like a good idea at the time, but no point of using it for a tiny campaign. Here's a script that will delete the tables. It's posted here because the InnoDB engine complains if you drop these in the wrong order, and violate some constraints.


drop table if exists `civicrm_activity`;
drop table if exists `civicrm_activity_history`;
drop table if exists `civicrm_activity_type`;
drop table if exists `civicrm_address`;
drop table if exists `civicrm_county`;
drop table if exists `civicrm_custom_option`;
drop table if exists `civicrm_custom_value`;
drop table if exists `civicrm_donation_page`;
drop table if exists `civicrm_email_history`;

For a while, I've bought into an idea that some folks (like Dave Winer) have promulgated about information. They say that there are three ways to organize information: chronologically, tabular, and hierarchically. (Notably, project management software combines all three.)

It seems about right, though the description seems too facile. Right now, computers can have tables and hierarchies, but there's not much support for storing and querying collected time-interval data. Most time data are treated as "atoms," when, in reality, most time data is some kind of interval.

One other kind of information seems to be missing: indeterminacy. Quantum mechanics (a subject about which I know little) and fuzzy logic (again, don't know much) both deal with the fact that you don't really know the value of a measurement, until you also know how accurate your measurement is. In other words, you need to know how much confidence you have in your data; statistics and statistical methods (a class I didn't do well in) are important.

Business

It's been a while since I've blogged, because of a short term overload of things to do, putting in full time at a client's, and also taking a business class at the CC. Yes, business. It was a little bit of a shock to my anarcho-socialist mind to learn that most of the ideas we hold aren't different from the ideas taught in biz school... it's the "masses" who are uninformed about it... and it's only the anarcho-socialistic types who think something is wrong with the way things are. The biz class attitude is to just get yourself on top of the pyramid, and hope to avoid a revolution. I suspect biz school is also a leading vehicle for teaching the finer points of shady business practices and political corruption, but I can't be sure of this.

What I was hoping for were lessons on accounting and standard business practices, negotiation, and laws. Being an intro class, though, none of that was being taught. So, I'll have to just take some specific classes for those things. What was taught: money, organization, government, marketing. It's just basic capitalism, and as enlightening as sitting through an intro to Marx, Weber, or Gramsci. The book was okay, but was biased in favor of business, and also had some factual errors. The factual errors kind of killed it for me.

This is a Perl version of the VB FileMirror class shown earlier on this site. The class helps you write scripts to (semi) safely mirror or move a file system from one directory to another. It does this by building the correct destination paths, making destination directories, and deleting files only after they appear to have been copied over.

Docs are in FileMirror.txt, attached.

DriveGMover.pl is a script that uses the class.

This is a first draft of some classes to manipulate .htaccess and htdigest files. I'm putting it up because there aren't many classes to do this, and seemingly none for htdigest files. (I just learned about the PEAR::Config classes. Neat.)

HTAuthFile is a superclass with a common writer. HTAccessFile is a very incomplete way to manipulate Apache .htaccess files. It is basically incorrect, because it doesn't support those <Scoping things> in Apache config files, and it doesn't take into account the order of directives, and it doesn't deal with using the same directive mulitiple times, but it will do for writing simple files to enable directory passwords. It's at times like this, I wish Apache used XML for the config format... but, this is the only time I've really had that feeling.

This is a class that will help you find files on the disk, without hitting the disk too much. It's a simplified unix "slocate" library. The first time you use it, it creates an index of all the files on the drive. (Subsequent uses update the file when a day passes.) The index is loaded into memory, and searched with regular expressions.

On an AMD Sempron 2000, the first search on 200,000+ files takes around 11 minutes. The second search takes around 2 seconds, and subsequent searches during a single run take around 1 second each. The first run builds the database, and the subsequent searches use it.

I thought there'd be a whole system to deal with the bad files, by now, but my attention shifted to understanding ESRI datasets. Spot checks on the copied data indicated that there were a fair number of datasets used, and they weren't copied over. The topic seems to be involved, and there's a lot to read.

Datasets are basically databases. They're referenced indirectly, via a driver of some kind, and the driver, rather than the app, manages the data files. I suspect there's going to be a serious problem with these, because I was unable to navigate to a missing dataset via the UI. I'd click on the "Set Datasource" button, and a dialog would pop up, but clicking around the interface never seemed to bring up any data.

Chances are, data sources need to be mounted at the workstation before I can navigate to them.

This blog entry doesn't discuss any code, because I ended up doing very little work on the software. I was busy running it, and the app tends to take over the UI, making programing difficult. For the most part, it functioned as expected, but a couple things changed in how I actually used it.

I added a text box to restrict scanning for MXD files to a subdirectory. This new feature allowed me to set the source root and destination root directories, and then type the subdirectory to be processed. The text box could also be left blank.

I used this feature a lot.

The main feature I stopped using was the scheduler. The software was so slow that it failed to use the network 100%. The point of having the program pause at night was to let backups happen faster, to avoid competing for the network.... but since the job appeared to be CPU-bound, I had was a big disincentive to be a good network citizen. The most efficient thing to do was let the app run all the time, even if it impacted the network -- every time the app was "nice" to the network, the length of time to complete it's task would grow by that amount plus the additional CPU time it took to complete the task.

It was looking pretty grim for the file copier. The main problems were twofold:

First, the app sometimes crashed on bad data. The Windows crash-reporting dialog box came up, and all work stopped until it was dismissed.

Second, after dismissing the dialog box, the app looped furiously, failing to process the remaining files, but marked them as completed (that's a little design bug there). The problem was that the COM server was returning an exception, and it was being trapped and effectively ignored. I say "effectively" because a special call to kill the app, via WMI, didn't clear the problem -- the underlying COM server didn't get stopped and unloaded.

The error could be cleared if I paused the batch job, and restarted it. That caused the thread that communicated with the COM server to abort, thus (probably) releasing the COM server. If only this could be automated.

I'm finally running that program I've been writing. For the most part, it was "bug free" inasmuch as the different parts ran their test cases correctly, and it runs fine on a small subset of data. Of course, it's not like I ever really learned the system completely, so there are a lot of situations that I'm not handling (or even aware of). Moreover, because the users probably don't use all the features of the software, it's not likely that a complete solution is necessary. No project is debugged "enough" until it operates on real data, in a real situation. Fortunately, my first iterations were done in Perl and VBA, in a scripting environment, on real data, so my experiences debugging those determined the overall structure of the code -- that is, the code deals with messy situations.

It's going to be a lesson about recursion. It's strictly beginner level.

The following code is a hack to alter a Drupal layout to use two columns. It does this without going into the Drupal code. Instead, it alters the layout of the content, using "float" and "width" to turn a one-column layout into two.

Doing this is pretty easy. First, you get a reference to the content area, then use that to get an array of DIVs within that area. All DIVs are returned. Each DIV is inspected to see if its class is "node". If so, the layout is altered.

<script type="text/javascript"><!-- 
var content = document.getElementById("content-left");
var nodes = content.getElementsByTagName("div");
var nodeNodes = new Array();
var i = 0;
var j = 0;
for(i=0;i<nodes.length;i++)
{
   if ( nodes[i].className.match('node') )
       nodeNodes.push(nodes[i]);

Here are three Mac-style busy indicators, or spinners, running at three speeds.

Things are going slow on the coding lately. There's just a lot of non-programming stuff to do. The stuff I'm covering in Learning VB is being run through its paces. It's mostly working, but there are still some things to work out. For example, sometimes, documents will load up, seemingly okay, but show an alert box error in the GUI. Then, they won't throw an OLE exception until the alert box is dismissed. The real fix might be to totally ditch the GUI and use the objects directly.... which sounds a little more complex to me, for some reason.

Second, there are still timing problems. One way to deal with those is to add some polling loops, like this. obj is a Collection of objects that get created over time:

Do 
    lastvalue = obj.Count
    System.Threads.Threading.Sleep( 2000 )