Careful File Copy

This area of the site was getting really chatty, so I've removed it from the Software book, and moved it under DIY notes. The remainder of the experience will be put on the blog. Consider all the material on this page obsolete.

Prologue

I was asked to help migrate a large batch of ArcMac GIS files to a new server. The problems: the files contain references to other files, and all those files must also be copied. Also, these files are mixed in with other files, not pertinent to GIS, on a single server. To manage growth, it's necessary to move the GIS files out.

Also, it's not a simple file copy. ArcMap can store the references in absolute or relative form. At this office they were stored as absolute paths, because that's more reliable. Thus, it's necessary to flip this bit to "relative," copy it over, and then re-flip it back to "absolute."

Due to the large number of files, and the slow speed of ArcMap, I decided to try and script the process. This sub-site details some of what I've learned in the process.

For a little more info: http://www.acadweb.wwu.edu/gis/tutorials/ArcMap_File_Mgmt.htm

Part 1

These are the product of the first iteration of this project. It succeeded in some ways, but failed in other ways. It succeded in processing around 150 files before it crashed and failed to process the remaning 1,400 or so.

I concluded that a longer-term project was feasible, due to the tedious and slow nature of this task. (That is, the tedium of copying files exceeded the tedium of reading hundreds of pages of VB help docs, which use "enterprise"-style code examples that usually don't do anything useful, as presented.)

The code

Taken together, these scripts form an almost-functional system. Some of the scripts are installed in an Excel document, and others are installed into the Normal.mxt template in ArcMap. The perl file copier script should be run as a scheduled task. It runs under ActivePerl.

The big problems, so far: VBA doesn't handle OLE server timeouts well; ArcMap chokes on some files; the scripts use the IMxDocument interface instead of the IMapDocument, which might be faster; the scripts pause for one minute while the mxd file loads, instead of polling the app to see if the file is loaded.

The small problems, so far: it'd be better to have the file copier written in VBA; the file format for the manifest files (generated by a script from the esri dev site) should be written for computer processing; using Excel as the process db is kind of cheesy.

Being a noob, I didn't realize that Microsoft's idea of "Automation" was not very thorough. OLE automation, as implemented with Excel and ArcMap, isn't stable enough to do real batch processing. With VB (not VBA) driving ArcMap, I suspect it's possible, but ArcMap will still not provide good error handling.

The code below has the following interesting features:

  • a process find-and-kill function
  • a recursive file system scanner
  • an excel-based tool to apply a macro to many files
  • a little error handling
  • mxd file analysis

References

The LayerSourceArray code from the ESRI dev site.

http://search.cpan.org/~jdb/libwin32-0.26/OLE/lib/Win32/OLE.pm

about using multiple interfaces via scripting (you don't)

http://support.microsoft.com/kb/193247/en-us?spid=2513&sid=946

http://www.microsoft.com/technet/scriptcenter/guide/sas_wmi_jgfx.mspx?mfr=true

AttachmentSize
sapphos.bas.txt1.29 KB
filecopy.pl.txt6.1 KB
FileBatcher.cls.txt2.82 KB
FileSystemScanner.cls.txt1.56 KB
main.bas.txt4.84 KB