Extended Attributes on Mac OS X and File Transfer

I recently encountered an interesting issue on Mac OS X. One of the applications I was working on, transfers files to a server. The transfer program first compresses the file using zip algorithms (zlib) and then sends the files to the server via a socket connection. Everything worked well except for some files. These files had extended attributes. HFS/HFS+ filesystems on Mac OS support extended attributes for files. These attributes are metadata that the application creator adds to the files. You can read more about extended attributes from the wikipedia link above.

The following commands to view the extended attributes on files -

$ ls -l@

The above command displays the extended attribute names (if they exist) for each of the files in the directory.

Another command that can be used is -

$ xattr <filename>

The above command can be used to list/add/delete extended attributes.

For some applications stripping off the extended attributes doesn’t make a difference to their functionality. But for others these are essential. Examples of some extended attributes are “com.apple.FinderInfo”, “com.apple.ResourceFork“. The FinderInfo is used by the Mac OS Finder.

Some applications don’t handle extended attributes well one such application being zip. Copying such files to file systems that don’t support extended attributes also result in loss of information. “rsync” has options to copy files with extended attributes. One other thing that is misleading is that these extended attributes don’t contribute to the file-size when viewed via a shell (ls command, stat command etc.). I could only see the difference when viewed via spotlight information. And the checksum of the file remains the same with and without the extended attributes.

However tar doesnt have a problem in keeping the attributes intact.

One way to circumvent the stripping of these attributes is to tar and then zip the files before transfer or write a program that uses getattrlist and setattrlist to handle these attributes separately.

Other useful references for extended attributes and acls -

http://forthescience.org/blog/2007/12/11/macosx-leopard-extended-ls/

http://www.macgeekery.com/tips/the_new_resource_fork

http://lists.apple.com/archives/Filesystem-dev/2009/jul/msg00011.html

http://lists.samba.org/archive/rsync/2007-March/017528.html

Configuring Eclipse as TCL/TK IDE

To configure Eclipse as a TCL/TK IDE you would need the plugin DLTK (Dynamic Language Toolkit). Information on DLTK is available http://www.eclipse.org/dltk/

DLTK supports other scripting languages such as PERL, PHP, TCL/TK etc.

Things you would need to setup TCL/TK Eclipse IDE -

• Eclipse 3.4 or greater (ofcourse) : Java or JavaEE IDE.
• DLTK’
• TCL/TK
• Komodo Remote Debugger

Steps to follow are -

• Download and install Eclipse 3.4 or greater from eclipse.org (install the Java or JavaEE IDE).

• To install the DLTK, in your eclipse IDE navigate the menu item “Help > Software Updates…”, or “Help > Install New Software…” to install plugins/updates.

• Add this as the update site for DLTK – http://download.eclipse.org/technology/dltk/updates-dev/1.0/

• After the site has been added it would show the list of plugins available under DLTK. Choose the following -

>>> Dynamic Language Toolkit – Core Frameworks

>>> Dynamic Language Toolkit – Core Frameworks SDK

>>> Dynamic Language Toolkit – iTCL Development Tools

>>> Dynamic Language Toolkit – iTCL Development Tools SDK

>>> Dynamic Language Toolkit – TCL Development Tools

>>> Dynamic Language Toolkit – TCL Development Tools SDK

>>> Dynamic Language Toolkit – XOTcl Development Tools

>>> Dynamic Language Toolkit – XOTcl Development Tools SDK

• After you have selected all the above packages and agreed to the license agreement, install them.
• After the install of the components, eclipse will re-start and the TCL environment (perspectives, windows etc.) would have been configured into eclipse.
• Now download and install TCL Shell and interpreter itself. For Windows, you can download it from ActiveState (http://www.activestate.com/activetcl/) and for Linux download it from www.tcl.tk/software/tcltk/
• After you have installed TCL in your preferred path/location; Open Eclipse IDE and follow the menu option “Window > Preferences > TCL“; Click on “Interpreters“, click Add button on the right, enter interpreter name as “TCL” and click “Browse..” and choose the path to the Tcl interpreter executable. Click OK and save the setting.
• Now you are done. You can write and run TCL programs via Eclipse.
  
• To debug your Tcl programs you would need to install the Komodo remote debugger.
  
• You can download Komodo from activestate – http://aspn.activestate.com/ASPN/Downloads/Komodo/RemoteDebugging
• Install the above software under a directory of your choice.
  
• Now, open eclipse, and navigate to the following menu option “Window > Preferences > TCL“, expand TCL and then choose “Debug > Engines > Active State“. Under the “Paths” tab, select the “Path:” dropdown and under “External Debugging Engine” select browse and choose the executable for komodo remote debugger (for windows it would be dbgp_tcldebug.exe). Click OK and save the setting.

Now you are all set. You should be able to choose new TCL project, write your Tcl code and debug your programs.

Writing C++ Applications for Versant

In this write up we shall see what does it take to develop applications using Versant’s Object Oriented Database. I have been involved in developing and debugging applications with Versant as the database for bit over an year and this write up is just a recap of the kind of work I have been doing with Versant using its C++ libraries. There are options to use other languages like Java, but I’ve used C++.

NOTE: This write up is only contains the approach with brief reference to its C++ API classes. It does not describe the C++ API in detail. For that you would need to get your hands on the Versant Usage Manual and Versant Reference Manual for C++.

Definitions

Firstly what is an Object Oriented database ?

Object Oriented database (a.k.a Object database) allows its user to represent information as Objects. The schema is created as a set of classes and the instances of the class are persisted into the database and are called objects. The advantage of using an Object database is its easy to represent classes you have designed into the database directly. You could use all the advantages of object oriented(-ness) for your database objects. You could design base classes and derive classes with more special functionality. And invoke virtual methods on the base-class (after loading them from the database) which could invoke a derived class functions. And use complex object composition to represent one object holding other objects.

(Also See: http://en.wikipedia.org/wiki/Object_database)

Some Basics of Versant

Before we begin developing applications we need to be aware of some concepts, classes, tools, and procedures that Versant provides its users/developers.

Concepts

Once you have your persistent classes ready (we will see how to get these classes later). You could stream the schema into the database. And then populate them with data (object instances).

Query Language

All queries to the versant database are written in a propreitary query language called VQL (Versant Query Language). You could refer to the reference manual to learn about this.

Internal Representation

Each object is uniquely identified by OID (Object Identifier). An OID is a numeric string that takes the form of “dd.nn.nnnn” where dd, nn and nnnn are integers. Example -

1.0.4941,

32.36.43204 … etc.

Concept of Transaction and Writes

When modifying the object instances in the database using the APIs, we could begin a transaction and start modifying the data, each of the data we modify we mark it as dirty. And at the end if the transaction is committed only that data which was marked dirty actually gets written to the database.

Templates and Prerequisites

Many of the Versant’s C++ API classes makes extensive use of Templates (parameterized classes). If you are not familiar with C++ templates you could just do a quick read up on the basics of it. As we would only use the already provided template classes it shouldn’t be that hard.

Types of Objects

Objects in versant can be

Persistent — those ones which were / can be  saved to the database.

Transient — those which only reside in memory.

Versioned Objects — Objects could be versioned. You could have a persistent object and create newer versioned objects based on the base/parent object. You could have a version tree of the objects.

What Versant Persistent Classes can Encapsulate?

Versant Persistent classes can encapsulate the portable data types that are shown below and other persistent classes, collection classes. However it cannot encapsulate enums, unions, and C style pointers or C++ references. If you want to indicate you want to refer to another class as a link, then you should use Link<type> classes of Versant.

Important Versant Classes

PObject : Is the base-most class. For any persistent Versant Object. This has methods  -

acquirelock — acquire a lock on itself,

as — method to down-cast to a more specific derived class

operator new — Creates a new persistent object for any sub-type. There are some helper macros for this like – O_NEW_PERSISTENT

delete — delete itself from memory and database based on the type of object it is (destructor is called).

releaseobj — release the object from memory.

PVirtual (inherits from PObject) : This class has methods for hashing and searching.

Visual Studio & Starteam Integration

If you have installed Starteam Client on your machine and if you are getting these annoying messages during startup of Visual Studio that says something like

“No Interface Found”  -or-

“<ProjectName> not found in Source Control” or something to this effect

And if you don’t intend to configure starteam as your source control with Visual Studio then just un-install “Starteam SCC Integration” package.

Configuring Eclipse 3.4 (Ganymade) with Subversion (Subclipse)

To configure Eclipse 3.4 CDT on Linux (Ganymade) with Subversion is easy and very handy during development.

If you don’t already have SVN you could install subversion RedHat based linux box using the command -

# yum install subversion

If you are on KDE there is a nice UI client available “kdesvn”.

To configure SVN integration with Eclipse install the subclipse plugin.

Open Eclipse and select “Help->Software Updates” then select “Available Softwares” tab.

Click on “Add site” and add the URL – http://subclipse.tigris.org/update_1.6.x

Click on the checkbox after which the following packages would appear which would need to be installed -

• Core SVNKit library
• Optional JNA Library
• Subclipse

After this there are perspectives and “SVN Repository Exploring”.

Note : You’ll have to ensure that the subversion server you are installing is not incompatible or too old compared to what client you have. Else you may see errors like these -

This client is too old to work with working copy ‘…’. You need to get a newer Subversion client, or to downgrade this working copy. See http://subversion.tigris.org/faq.html#working-copy-format-change for details.

Build Apache 2.0.x + Configure PHP with Apache

To build apache, download the apache sources from Apache site. Unzip apache, run the standard configure, make and make install commands.

To configure PHP with the above build apache server -

$ ./configure –with-apxs2=<Apache Install Directory> \

–with-mysql  –prefix=<PHP Install directory> \

–with-config-file-path=<PHP Install directory> –enable-force-cgi-redirect \

–disable-cgi  –with-zlib –with-gettext  –with-gdbm

$ make

$ make install

Awesome PHP Configuration resource – http://dan.drydog.com/apache2php.html

MySQL configuration and startup

To configure MySQL with Auto-Start on Linux -

>> run the following command as root/superuser

# mysql_install_db

>> To configure mysql for automatic startup during boot time

# cd /etc/rc.d/rc3.d

# ln -s /etc/rc.d/init.d/mysqld S98mysql

# mysql start

>> Tool to secure your MySQL installation run the following script as non-root user.

>> Read through each of the questions and choose an appropriate option

$ /usr/bin/mysql_secure_installation

StarTeam ANT Task Extension (Check-in with Task ID)

Apache Ant has StarTeam tasks for check-in, check-out, label etc. I’ve modified the StarTeam Ant task to check-in with a specified Task ID. This could easily be extended to a CR (Change Request) as well.

The code (ant-starteam-ext-src.jar) and JAR (ant-starteam-ext.jar) has been shared on my box.net share. Look for my Box.Net share on my blog’s home page.

Some of useful links for developing ANT tasks for StarTeam -

http://techpubs.borland.com/starteam/2008r2/en/SDK_Documentation/api/index.html (StarTeam 2008 SDK)

Look for the ANT API in your ANT_HOME/docs/manual/api folder.

Building Visual Studio 2005 Project via Apache Continuum

Configuring Visual Studio 2005 Builds with Apache Continuum gives the error “fatal error C1902: Program database manager mismatch; please check your installation”.

Visual Studio 2005 has a bug for builds that are scheduled using the AT command or scheduling tools. This has been fixed via patch provided at the Microsoft KB920770 link. Apply this patch and try re-scheduling the build and it would work fine.

Build Instructions : Xerces 2.7.0 and Xalan 1.10 (64-bit on x64) using VS 2005 / VC 8

Building a 64-bit Xerces 2.7.0 and Xalan 1.10 is a set of additional steps to “Build Instructions : Xerces 2.7.0 and Xalan 1.10 (32-bit) Using Visual Studio 2008 / VC++ 8″.

Read my blog on that for full instructions to build them using 32-bit settings.

Xerces 2.7.0 Build Instructions

And for building the Xerces 2.7.0 in x64 mode perform the following steps -

(0) Open the xerces-all.sln file under VC8 folder (that was created under 32-bit build).

(1) Add new configuration setting; You could name it something like — “Release_x64″. Copy settings from “Release”.

(2) Add a new platform “x64″ also have been added. Copy settings from Win32.

(3) For the “Release_x64″ <–>  “x64″ configuration setting, make sure you change all build output paths to win_x64 from Win32.

(4) Goto project properties dialog of XercesLib project. Here add “Win64″ as a pre-processor definition. This is needed for compliation to go through successfully in 64-bit mode.

(5) Now compile the XercesLib project. This should compile without any errors and generate the dlls required.

Output files are generated under XERCESCROOT\build\win_x64\vc8\release

For 32-bit builds use the “Release” configuration with Win32 platform settings.

Xalan 1.10 Build Instructions

To start with open the  Xalan.sln – Create new Configuration settings “Release_x64″ copied from Release and “x64″ machine setting copied from Win32.

Under XALANROOT\Projects\Win32\VC8\Utils under the 3 folders “Localization”, “MsgCreator” and “XalanMsgLib” make changes to the .mak files to incorporate x64 build and build settings. If you would like to download the entire package of Xalan 1.10 sources modified for x64 settings in the way I’ve described here, then you could pick it up from my box.net share.

NOTE: Original Win64 settings in Xalan caters to IA64 build, I have changed it so that it can be used for x64. Alternatively you could create a new setting altogether for x64 in which case you would have to modify things in many other places like visual studio settings etc.

Build only the localization project from visual studio. This should create all the necessary files under

XALANROOT\build\win_x64\vc8\release

In “AllInOne” project change the build settings (C++ and Linker) sections to make it point to Win_x64 directories instead of Win32.

Then build “AllInOne” project. This should then create the xalan libraries in 64-bit mode.

NOTE – Take a look my box.net’s widget and download the xerces 2.7.0 and xalan 1.10 which can be compiled in both 32 and 64 bit configurations.