How does Piper fit into the repository ecosystem? It is a behind the scene tool for repository administrators to curate collections. For our initial phase we focused on the sole task of ingesting content into DSpace simple. However, in the future, it may bring in workflow capabilities to ensure quality control, integration with other workflow tools like Vireo, and of course additional repository support.

How does Piper work?

First a content owner works with a librarian to gather their content together. The metadata is captured on a spreadsheet, and all the files are collected into a folder. The folder is copied to shared network storage that both the librarian has access to on their desktop and the server running Piper can access. We chose network file sharing because batches can be bigger than the 4GB limit that the HTTP protocol imposes. In the future,  a direct upload mechanism for small batches may be added primarily for convenience. Next, the librarian logs into Piper through their web browser and uploads the spreadsheet with the metadata for the batch. Currently, the only metadata format supported is a Dublin Core spreadsheet where the first column is a reference to the file, and all other columns are a Dublin Core field specified in dot notation. Piper is designed to be easily extensible so each “Ingester” is a simple Spring bean. When uploading a new spread sheet the user selects what format to use.

Once a batch has been ingested into Piper, it is ready to be verified. The current version is read-only, i.e. to make any changes you need to go back to your spreadsheet update it and re-upload it. However, the necessary UI adjustments to edit the content directly with-in Piper will likely be added in a future version. Each repository is pre-configured with a set of verifiers (each is a simple Spring bean for easy extensibility). Each verify will scan the batch and identify any errors – things that we know will not work, or warnings – things that might not work. Some things they check are whether the column label is valid Dublin Core, or whether 90% of the items contain a particular field but there is an item with that field missing. Right now there are seven different verifiers almost all of them are completely related to DSpace and Dublin Core. As other formats and repositories are supported the list of verifiers will expand.

Once a batch has been verified and contains no errors it is ready to be deposited. The librarian simply clicks the deposit button and each item is ingested into the repository one at a time via SWORD into the destination collection. Right now Piper only supports SWORD version 1, so it’s a one shot deal. With support for Sword 2 in a future version, it will allow us to deposit the batch into a repository and then some time later if we find an error re-deposit the item back into the repository updating the existing item in the repository without assigning it a new identifier.

Technical notes

The application is developed using the Play Framework using Java, Spring, JPA, etc. These collections of tools proved to be a very productive environment for Piper. One of the cool new HTML 5 technologies that we used in Piper is Web Sockets. Web sockets allow the browser to open a persistent TCP connection back to the server for two way communication. We use this extensively in Piper for the spreadsheet view. When the page loads initially the content is blank. The browser opens a web socket back to the web server where Piper starts feeding all the batch content over the web socket back to the browser for display. This allows us to display a page with 20k items on it and not deal with time out issues, or blocking issues. The page is constantly working in the background to display the items while the user is able to interact with their web browser. This approach also allows us to push updates to the users browser. This is important because some of the tasks are long running such as verifiers, ingesters, depositors, etc, may take several minutes to hours to complete. While these are running on the server the user’s browser is able to update the changes in real time without reloading the page. Very cool stuff.

What is next?

Piper is an internal project for Texas A&M University Libraries and has not gained approval for release as an open source project. I hope that we will be able to make it public so that others can check it out and see if it’s useful for them. Right now it only supports a very limited set of formats that need to be expanded. Most of our development focus for the rest of the year will be on Vireo, so it may be sometime before we jump back to further development on Piper.

Screenshots

What happened?

Our documented upgrade procedures calls for us to copy of the Solr directory from the old instance into the fresh instance. However I typoed the command so that the old copy of the Solr directory ended up inside the fresh copy of the Solr directory. Then we didn’t catch the error for a few weeks. So on the repository statistics page we were only showing stats from when the upgrade occurred, all the other months were zeroed out.

Opps!

How to fix it?

If you found your self in a similar predicament then you can recover. The import thing is that you have both your old statistics data and your new statistics data. You just need them combined into one data set. DSpace uses Solr (which is built upon Lucene) for storing statistics information. Because of this you have two basic approaches one at the Solr level and the other at the Lucene level. The basic concept is that you need to merge the two indexes together. The solr wiki describes these two methods. At first I attempted down the Solr path but I ran into a road block early when I was unable to issue the Solr command to create a new core where the two merged indexes would reside. Then I tried the other option and the Lucene MergeTool worked well. Here are the steps I followed to restore statistics.

Step by Step Instructions

1) Identify DSpace’s copy of the Lucene libraries: lucene-core and lucene-misc.

You will find these inside DSpace’s solr webapp: <solr webapp>/WEB-INF/lib/lucene-*.jar For DSpace 1.7.x the version for these libraries were 2.9.3. It is important that you use the same Lucene version that wrote the original indexes. We’ll use both of these paths below in step 5 as <path to lucene-core jar> and <path to lucene-misc jar>.

2) Identify both Lucene indexes.

Typically the solr-based statistics index is stored inside the DSpace install directory: <dspace directory>/solr/statistics/data/index. Inside the directory you should see at least one “.cfs” file along with a “segments.gen” file. The other copy will likely come from a back of copy of DSpace you have stashed away somewhere. We’ll use both of these paths below as /path/to/oldindex1 and /path/to/oldindex2.

3) Shutdown your DSpace instance.

The merge tool requires that all the indexes it is reading be closed so while the merge is processing you can not be recording any new statistics.

4) Run the Lucene merge tool.

java -cp <path to lucene-core jar>:<path to lucene-misc jar>
    org/apache/lucene/misc/IndexMergeTool /path/to/newindex
    /path/to/oldindex1 /path/to/oldindex2

The command above will merge both the old index into the new index. The command takes about the same amount of time as it does to copy both indexes.

5) Restore the combined index.

mv <dspace directory>/solr/statistics/data/index /path/to/someplace/safe
cp -r /path/to/newindex   <dspace directory>/solr/statistics/data/index

6) Restart DSpace and check the statistics.

Hopefully everything works and you have a full set of statistics. Let others know that it worked in the comments.

It’s actually quite hard to find the absence of something with SQL. After trying various methods the way I came up with to solve this problem is a sub select that counts how many anonymous access policies exist for each object and if there are none then report those. The query is broken down into three distant parts one for each object time. Then all the objects are combined via PostgreSQL set operators and sub selects (again!). This means that if you have a huge number of restricted items in your repository the query might fail or take an obscene amount of time/memory to run. I tried using a left outer join but couldn’t get it to handle the case where both no access policies exists and only non anonymous access policies exist.

The approach used here is inelegant and has some serious performance problems. However it worked my immediate purpose. We had no idea how many or which items are restricted in our repository (answer: just under 300). This task is a good candidate for a DSpace curation task, to find all items in a collection which are have restricted access. Or the opposite, find all items which are NOT restricted.

Here is the code for you to cut and paste:

SELECT DISTINCT ON (handle) handle
FROM handle
WHERE 

handle IN
(
-- ---------------------------------------
-- Select Items which are restricted
-- ---------------------------------------
SELECT DISTINCT ON (handle) handle
FROM item
INNER JOIN handle
ON item.item_id = handle.resource_id AND handle.resource_type_id = 2
WHERE
(
-- Count how many anonymous access policies exist for each item
SELECT count(*)
FROM item AS item2anonymous, resourcepolicy AS rp
WHERE
item2anonymous.item_id = item.item_id AND
item2anonymous.item_id = rp.resource_id AND
rp.resource_type_id = 2 AND -- Type   = Item
rp.action_id = 0 AND        -- Action = Read
rp.epersongroup_id = 0      -- Group  = Anonymous
)
< 1
) 

OR handle IN
(
-- ---------------------------------------
-- Select Bundles which are restricted
-- ---------------------------------------
SELECT DISTINCT ON (handle) handle
FROM item2bundle
INNER JOIN bundle AS bun
ON bun.bundle_id = item2bundle.bundle_id
INNER JOIN handle
ON item2bundle.item_id = handle.resource_id AND handle.resource_type_id = 2
WHERE
(
-- Count how many anonymous access policies exist for each bundle
SELECT count(*)
FROM bundle AS bun2anonymous, resourcepolicy AS rp
WHERE
bun2anonymous.bundle_id = bun.bundle_id AND
bun2anonymous.bundle_id = rp.resource_id AND
rp.resource_type_id = 1 AND -- Type   = Bundle
rp.action_id = 0 AND        -- Action = Read
rp.epersongroup_id = 0      -- Group  = Anonymous
)
< 1
) 

OR handle IN
(
-- ---------------------------------------
-- Select Bitstreams which are restricted
-- ---------------------------------------
SELECT DISTINCT ON (handle) handle
FROM bitstream AS bit
INNER JOIN bundle2bitstream
ON bit.bitstream_id = bundle2bitstream.bitstream_id
INNER JOIN item2bundle
ON bundle2bitstream.bundle_id = item2bundle.bundle_id
INNER JOIN handle
ON item2bundle.item_id = handle.resource_id AND handle.resource_type_id = 2
WHERE
(
-- Count how many anonymous access policies exist for each bitstream
SELECT count(*)
FROM bitstream AS bit2anonymous, resourcepolicy AS rp
WHERE
bit2anonymous.bitstream_id = bit.bitstream_id AND
bit2anonymous.bitstream_id = rp.resource_id AND
rp.resource_type_id = 0 AND -- Type = Item
rp.action_id = 0 AND        -- Action = Read
rp.epersongroup_id = 0      -- Group = Anonymous
)
< 1

)

]]> http://www.scottphillips.com/2011/08/find-all-restricted-items-within-dspace/feed/ 0 http://www.scottphillips.com/2011/07/character-encodings-dspace-excel-os-x/ http://www.scottphillips.com/2011/07/character-encodings-dspace-excel-os-x/#comments Wed, 20 Jul 2011 14:41:38 +0000 Scott http://www.scottphillips.com/?p=977 The problem I recently ran into was updating the metadata for a particular collection that was being moved from TDL’s repository into A&M’s repository. I able to quickly move the collection into the new repository using OAI-PMH harvesting with ORE support. However, the metadata needed a bit of cleaning up for it’s new repository home, such as changing dc.contributor.author to dc.author and inconsistent formats used in other fields. This is a perfect task for Stuart’s Bulk Metadata Export tool. This DSpace feature allows an administrator to download a Comma Separate Values (CSV) file of all the metadata in a particular collection, then open it up in MS Excel and edit the metadata naturally. Finally once the metadata is ready to go you can upload it back to the repository and all the fields will be updated correctly. It is a very nice feature that can save a ton of time.

The Problem

When I opened the file in Excel some of the characters were not showing up correctly. Specifically characters in titles and names which used non-English marks, in this case there were all from the extended Latin character set. If you ignore these problems, later when you try to upload the CSV file DSpace will pick up on these changes and cause the garbled characters to be introduced into the repository.

DSpace uses UTF-8 file encoding for everything as does almost all well-behaving application out there. Somewhere between downloading the file and opening it with Excel the encoding is being mis-interpreted. Excel has a CSV import tool where you can specify the character encoding but that tool does not work with DSpace’s export because the columns become miss-aligned if there are any new lines or charge returns in the metadata.

The Solution

The solution to this problem turned out to be using a command line tool I was not familiar with: “iconv“. It’s apparently a standard Linux command that is available with the default install of OS X. After some experimentation I found that when you “double click” open a CSV file Excel wants the file to be encoded using the default encoding found on Windows machines: Windows-1252. This encoding is very similar to ISO-8859-1 but has some key differences. Use the iconv command to convert the file into this encoding and you should be able to open the file with Excel OS X with just a double click. All the characters should be preserved.

iconv -f UTF-8 -t WINDOWS-1252 original.csv > excel.csv

However, you can’t upload this encoding back to DSpace because it won’t know what to do with it. DSpace expects it to be in the standard UTF-8 encoding. Once you are finished editing the file to your satisfaction in Excel you will need to convert it to UTF-8. When you save the file in Excel you will have two valid formating options you may select from the entire list shown below: “Comma Separated Values (.csv)” or “Windows Comma Separated (.csv)“. The key difference between these two formats is what character encoding Excel will use when saving the file.

If you went with the default Comma Separated Values format then you will need to run following command to convert the file back to UTF-8.

iconv -f MAC -t UTF-8 excel.csv > upload.csv

However if you chose the other valid option and saved the file as a “Windows Comma Separated (.csv)” then you will need to use the alternate command to convert the file back to UTF-8:

iconv -f WINDOWS-1252 -t UTF-8 excel.csv > upload.csv

You are ready to upload the resulting “upload.csv” file to DSpace or use command line tool for bulk editing.

There are already many tools available that do this task [1][2][3], however they all require the use of SVN commit-hooks. Commit-hooks are run on the repository’s server allowing external tools to be notified when specific events occur. Using commit-hooks can work reasonably well if you have access to the server’s configuration, but that is not always the case. Instead of relying on commit-hooks; the SvnBot runs independently, periodically polling the repository for any updates. When an update in found a message will be announced in IRC.

Below is an example IRC message. The message includes the author and their commit message along with some brief statistics about the number of files affected and a URL. If multiple files were affected by a single commit then the url reported is to the common path, i.e. the closest directory that contains all the affected files.

Scott: “SAND-30, reviewed the pom file: removing unneeded dependencies, declaring output to be UTF-8 and added a few comments.” (Rev 666: 1 file modified) https://texasdl.jira.com/svn/SAND/svnbot/trunk/pom.xml

Special thanks to the Texas Digital Library, my employer, for allowing me to release this as an open source project.

Download

• SvnBot 1.1 – Java Binary Distribution (ZIP)
• SvnBot 1.1 – Java Source Distribution (ZIP)
• SVN Repository

How to use

From the binary distribution simply run the Java jar file with the “-c” parameter passing the path to a configuration file. An example configuration file is provided with the with both the binary and source distributions.

java –jar svnbot-1.1-jar-with-dependencies.jar –c path/to/config

Since many will want to deploy SvnBot on a server and forget about it, you’ll probably want to run SvnBot within a nohup operator. The nohup command will allow the Java process to continue running even after you’re shell has disconnected (i.e. ignore the “hang up” signal). Here’s an example as for quick reference:

nohup java -jar svnbot-1.1-jar-with-dependencies.jar -c path/to/config
      < /dev/null >> /dev/null 2>> /dev/null &

Configuration

The configuration file format is a mix of XML and standard Java properties similar to Apache’s httpd configuration format. There are three types of configuration sections: <irc>, <log4j>, and multiple <repository> tags.

<irc> This section configures the IRC server and channel where to announce changes.

<log4j> This section configures how logging messages from SvnBot will be handled. This uses standard log4j syntax.

<repository> This section configures an SVN repository to monitor: specify the url, authentication, polling frequency, etc. The SVN bot is capable of monitoring multiple repositories, just add multiple definitions for each repository.

Below is the example configuration file. Note only the IRC’s server, channel, and repository’s url are required parameters. All other parameters are optional, either for special circumstances or have reasonable defaults.

# SvnBot Configuration
#
# java -jar svnbot-1.1-jar-with-dependencies.jar -c path/to/this/config
#
# Note you must use XML escaping such as: & < > &quot &apos

<irc>
    # The hostname, port, and channel of the IRC server
    server = irc.freenode.net
    channel = #svnbot
    #port = 6667

    # The nickname to use when communicating on IRC
    #nick = SvnBot

    # Authentication credentials for the IRC Server
    #username = your-irc-username
    #password = your-irc-password

    # The notice message is sent to everyone in the channel when the bot
    # first starts, letting others know that the bot has joined the
    # channel.
    #notice = I am an SVN Bot. I report commits to the ??? repository.
</irc>

# Optional logging configuration.
<log4j>
    # This section uses Log4j configuration syntax. The example below
    # will log messages to /var/log/svnbot.log. However, if no log4j
    # section exists then all messages INFO and higher will be sent to
    # stdout.
    #
    # See http://logging.apache.org/log4j/1.2/manual.html for more
    # information

    # Log everything INFO and greater to a File
    #log4j.rootLogger = INFO, FILE

    # Define the FILE's location
    #log4j.appender.FILE = org.apache.log4j.FileAppender
    #log4j.appender.FILE.File = /var/log/svnbot.log

    # Define the FILE's layout
    #log4j.appender.FILE.layout=org.apache.log4j.PatternLayout
    #log4j.appender.FILE.layout.ConversionPattern = \
              %d{ISO8601} [%t] %-5p %c %x - %m%n
</log4j>

# Repeatable repository configuration.
<repository type="svn">
    # The URL of your SVN repository
    url = https://yourepository/

    # Authentication credentials for the SVN Server
    #username = your-svn-username
    #password = your-svn-password

    # Targeted paths restrict the SvnBot's focus to only monitor changes
    # within the specified paths. There can be multiple paths separated
    # by a comma and each path must be anchored to the root of the
    # repository. Only commits to the repository that affects a file under
    # one of these paths will be reported in IRC.
    #targetPaths = project1/ project2/subproject, ...

    # The amount of time between polling the repository. Periodically
    # SvnBot will check the repository to see if there have been any
    # new commits. The default interval is 30 seconds. However, if
    # your repository is under heavy load you should use a higher
    # interval such as 60 or 150 seconds.
    #pollinterval = 30

    # Nicknames are used to translate the repository's usernames into
    # something more friendly. The username on the left side will be
    # replaced with the nickname from the right side when the commit
    # is announced in IRC.
    <nicknames>
        william = Bill
        charles = Chuck
        biglong@emailaddress.com = Short &amp; Sweet
    </nicknames>
</repository>

# Repeat the repository section for each repository.
<repository>
    ...
</repository>

How to compile

Prerequisites:

• Java 1.6 or higher
• Apache Maven 2.2 or higher

This project uses Apache Maven as its build and distribution management system. To compile the project use the following maven command:

mvn package

Two binary files will be compiled into the “/target” directory. You will find “svnbot-<version>-jar-with-dependencies.jar” which is a runnable jar file. The other jar file included is “svnbot-<version>.jar” which just includes the svnbot class files without other dependencies for SVN or logging. This jar file is suitable for inclusion in other projects.

This project uses the PircBot library to handle communication with the IRC Server. Another important dependency is the pure Java implementation of the SVN protocol with the SvnKit library.

References

[1] “SvnBot – Miscbits“, A Python SVN Notification IRC bot, using commit-hooks.

[2] “Writing an irc bot for svn commit notification“, A simple Perl script to send notification of SVN changes to IRC, also using commit-hooks.

[3] “IRCcat“, A Java-based IRC bot enabling shell scripts to send messages to IRC through a local port. This can be combined with a commit-hook to announce changes in IRC.

property.1.url = http://...
property.1.username = Bob
property.1.password = Bob’s secret

property.2.url = http://...
property.2.username = Joe
property.2.password = Joe’s secret

This way works but it’s sort of annoying and can be confusing for someone else to understand what’s going on. They would likely need to read the documentation, especially if it’s more complex with multiple types of repeating parameters. There are several alternatives, you could try encoding all the parameters into one property but that’s even harder for a user to figure out. A slightly better alternative is to use something hierarchical like XML, thus:

<properties>
	<repeatable>
		<url>http://...</url>
		<username>Bob</username>
		<password>Bob’s secret</password>
	</repeatable>
	<repeatable>
		<url>http://...</url>
		<username>Joe</username>
		<password>Joe’s secret</password>
	</repeatable>
</properties>

This is easier to understand, but it’s very verbose. Each property is labeled twice, once to open the tag and again to close the tag. XML is good for complex things like HTML or specific file formats with a dedicated reader. However, XML is not great for humans to read, let alone edit quickly.

A better solution, combine both!

Instead of either XML or properties file we can munge the two together to create something that is easier for users to manage.

property.one = value1
property.two = value2

<repeatable>
	url = http://...
	username = Bob
	password = Bob’s secret
</repeatable>

<repeatable>
	url = http://...
	username = Joe
	password = Joe’s secret
</repeatable>

The combined format is similar to Apache’s httpd configuration format where name/value pairs are also mixed with nestable elements. It’s very close to the simplicity of a plain old properties file, but has just enough expressivity to handle grouping of elements. It’s a win-win.

How to parse the mixed format?

It’s easy, thanks to the Apache’s Common Configuration project! Here’s a simple class to demonstrate parsing the combined file format. The full class can be downloaded here. Below I will walk through the important bits that make processing the config file easy:

private final static String PREPEND = "<?xml version=\"1.0\"?>\n<xml>\n";
private final static String APPEND = "\n</xml>";


List<InputStream> inputStreams = new ArrayList<InputStream>();
inputStreams.add(new ByteArrayInputStream(PREPEND.getBytes()));
inputStreams.add(new FileInputStream(configFile));
inputStreams.add(new ByteArrayInputStream(APPEND.getBytes()));

InputStream combinedInputStreams = new SequenceInputStream(
	Collections.enumeration(inputStreams));

This is a little trick to makes the configuration file a well formed XML document. It pre-pends an XML declaration and root level element to the beginning of the document while closing it at the end. All these parts are combined together using a SequenceInputStream which just reads from the first inputstream before moving on to the next on the list until all the inputstreams have been exhausted. Next we move on to parsing the configuration file.

XMLConfiguration xmlConfiguration = new XMLConfiguration();
xmlConfiguration.setDelimiterParsingDisabled(true);
xmlConfiguration.load(combinedInputStreams);

Next, we simply read the configuration file as a normal XML-based configuration file using the standard Apache commons library. One thing to note is that before reading in the configuration file it is a good thing to disable DelimiterParsing which is enabled by default. Normally the configuration parser will separate multiple values separated by a comma into a list, but since at this level each value is really a sub-properties file it gets in the way if a comma appears anywhere in the document. Next we move on to reading the root-level properties.

String rootConfig = xmlConfiguration.getString("");
PropertiesConfiguration rootProperties = new PropertiesConfiguration();
rootProperties.load(new ByteArrayInputStream(rootConfig.getBytes()));

Here we take all the text that is at the root of the configuration file (aka. Everything that is not inside a tag) and parse that as a properties file. Similar to how we read in the XML portion, this time we just pass all the text into the Apache’s PropertiesConfiguration object. From that object you can get all the properties parsed for strings, numbers, lists, etc by there names. After this point it’s just as simple as processing a properties file. Lastly, we move on to show how to process the repeatable sub-sections.

List<String> repeatableConfigs = xmlConfiguration.getList("repeatable");
for (String repeatableConfig : repeatableConfigs) {
	PropertiesConfiguration repeatableProperties =
		new PropertiesConfiguration();
	repeatableProperties.load(
		new ByteArrayInputStream(repeatableConfig.getBytes()));

	// Do something...
}

Here we get as a single list all the nestable elements with the name “repeatable”. From there we do the same thing as before, hand them to the PropertiesConfiguration object to parse the properties. It’s all pretty simple.

Thoughts

This is an elegant solution that forges a compromise between the expressiveness of XML and the simplicity of a plain old Java property file. This method is easy to implement and feels natural to any system administrator who’s use to editing configuration files on a regular basis.

There is one gotcha though, since we process the document as if it were an XML document all XML entities must be escaped. So if you need to include &, <, or > then you’ll need to escape them with “&”, “<”, or “>” or else you’ll create parsing exceptions.

Download the full class: ConfigurationExample.java

The domain expiration process is not always predictable and may take up to 80 days past the actual expiration date listed for the domain. While the domain is going through this process it will traverse through several states before actually being deleted.  Then once a domain is deleted from the system it becomes available for anyone to register it during the “drop period”. There are a hand full of companies that specialize in catching these freshly deleted domain names during this period. For this post I will start by describing the life cycle of a .com domain name. Next I will discuss the services that specialize in catching dropped domain names. Finally, I will conclude with a few thoughts of my own about the process.

The Life-Cycle of a .com Domain Name

The protocol used by the .com top level domain is the Registry Registrar Protocol (RRP) [4][5] which is also used by the .net extension along with the country codes. The RRP describes the possible states and some policy for handling expiring domain names. Other top level extensions such as (i.e .biz, .info, .name, etc..) use a newer protocol called Extensible Provisioning Protocol (EEP) [6]. Just a few days ago (January 2011) the owner of the .com extension, VeriSign, has agreed to switch to the EEP protocol [7] but as far as I can tell there is no time frame for the migration. If you’re looking at a domain name with an extension that uses EEP than this information may be incorrect.

The first question you probably have is when will the domain name be available? The process is not speedy and current domain owners are usually given a long grace period to renew their domain name. Even in the case where the current owner has no intention of renewing the domain name there typically is no way for them to tell their registrar that, it still has to go through the process.

After a domain name has reached it’s expiration date the current registrar is in the driver seat. Technically once the domain has expired it is their property. Luckily for the owner, registrars often allow the owner to renew the name during the grace periods. For all the reputable registrars the process takes 80 days: 45 days for Registrar Hold + 30 days for the Redemption Period + 5 days while Pending Deletion. However some of the less scrupulous registrars will change the time periods to their advantage.

Active:
This is the default status for a domain once it has been registered. The domain may be modified and will actively support a website, mail, or other services.  The owner holds the rights to the name and may renew the registration indefinitely or transfer the name to another registrar.

Registrar Hold:
After a domain has reached it’s expiration date it typically transitions into the “Registrar Hold” state at the discretion of the registrar. The registrar may keep the domain in this state for anywhere from 0 to 45 days. During this time the registrar is free to implement any number of policies as laid out in their individual terms of service [8][9][10][11][12]. Registrars have several options at this point including keeping the domain for themselves, selling or auctioning off it to another person, or letting it expire. The only thing that ICANN will force them to do is if they are going to let it expire it must go into the Redemption Period. However, most reputable registrars will allow the original owner to renew the registration with no extra fees during this period.

Redemption Period:
After a domain has expired before it can be deleted ICANN mandates that it must go through a 30-day redemption period. During this period the domain name will no longer be in the zone file meaning the website, email, or other services will no longer work for the domain. During this period the original owner may renew the registration for the domain, however there is a penalty fee.

Pending Deletion:
After a domain has gone through the redemption period it is moved to the pending delete status. This status lasts 5 days during with the domain may not be modified, nor can the original owner renew its registration. At the end of 5 days during the time period between 11am and 2pm Pacific Time the domain will be deleted, typically called the “drop period”. Once it’s deleted it is available for anyone to register it.

Once a domain has entered the pending delete status the original owner has lost all chances of renewing the registration. If they want to renew the domain name they will be on equal footing with everyone else attempting to register it during the drop period.

There are a few other states a domain may be in but they are all for abnormal situations such as legal action associated with the domain name. They are: Registry Lock, Registrar Lock, and Registry Hold.

Catching a Dropped Domain Name

A fleet of companies have specialized in acquiring deleted domains during the drop period. These aftermarket companies have researched the precise timing and little tricks to get any advantage they can during the drop period. They’ve invested in their infrastructure to have the available bandwidth to repeatedly query the DNS system to see when a domain is available, but not too much so that they get banned. Most of these companies work on a close auction model. This works by potential buyers placing minimum bids on the domain prior to a domains deletion. Then if the company acquires the domain it will go on auction with only those who placed bids previously being able to participate.

Many of the individual registrars have formed exclusive partnerships to handle aftermarket auctions for their expired domain names. Instead of dropping the domain for anyone to grab the domain is handed over to the partner company who will auction it off to the highest bidder.  Unfortunately it is hard to find out which registrars have exclusive partnerships [13][14], and they seem to change often [15][16].  I am not aware of any good rule you can use to find out which drop catching service you should use. Your best bet is to search the current registrar’s website to see if they explicitly state whether they have an exclusive arrangement with an aftermarket company or not. Sometimes it’s buried in their terms of service, other times they have made a press release announcing the partnership. However, If you can’t find this, then your best bet is to place initial bids with them all. That way you’ll still be in the game no matter which service catches the domain.

Pool.com

Pool.com pioneered the current model where buyers only pay if they acquire the drop with out any upfront fees. Now days this is pretty much the standard across the board. The minimum bid price for pool.com is $60. Multiple people may place bids on the same name. The system will not tell you if anyone else is bidding against you until Pool.com acquires the drop. Then, if multiple people placed bids the domain name goes to a closed eBay style auction.

SnapNames

SnapNames use to be the exclusive auction site for Network Solutions domains. The site is still going strong after that partnership ended. However, there has been a bit of scandal at the company where an employee was caught placing shill bids on domain auctions [17]. It looks like the company took appropriate action when it found out about the problem, refunding buyers money. However it is still a big black eye for the outfit, especially one in a business where trust and fairness is important.

The minimum bid with SnapNames is $79, and you do not need to pay unless you win the auction. In fact, you don’t even need to provide SnapNames with payment information until after you win. If SnapNames catches the drop then it goes into a 3-day auction between only those who placed a bid on the domain name prior to the drop. The auction is run as a typical eBay style auction where you can place multiple bids, set a maximum bid, etc. If no one else is interested in the domain then you’ll automatically win the auction at the minimum bid.

SnapNames caught the drop for my domain. The site will not tell you if anyone else placed bids on the domain name until after they acquire the drop. When that happened I found out that 9 other people also placed minimum bids on my domain! However the vast majority of those didn’t place a bid higher than their initial minimum bid. The bidding for my auction was obviously just between other Scott Phillips’s. I don’t believe the normal “domainers” were involved in the auction.

NameJet

NameJet is the newest company to the market, being formed when the partnership between Network Solutions and SnapNames ended. NameJet is the exclusive aftermarket auction site for Network Solutions, eNom, and Bulk Register [18]. If the domain you are interested in currently registered with one of these registrars then NameJet is the only site you need to deal with.

The minimum bid price starts at $69 and as normal you do not have to pay unless you win the auction. They offer both public auctions where anyone can place a bid, or private auctions where only those placed a bid before the drop can participate. I am not exactly sure how they decide which auction system is used, but I expect that the public auction is only used for the really popular domain names. Of course, if no one else is interested then you’ll automatically win the auction at the minimum bid.

GoDaddy

GoDaddy’s website is truly amazing in how hard it is to navigate. I can’t possibly imagine creating a more confusing experience. With their system you first purchase a “DomainAlert Pro Backordering” slot for $20.99. This slot can be used again and again until a backorder is successful. Once you’ve purchased that slot you can select which domain name you want to place a back order on. One thing to note is that they only allow one back order per domain name. So if, after purchasing the slot, it gives you a confusingly generic error when placing the back order it may be that someone else already has the one. As your consolation prize you can re-use the slot on another domain name.

If GoDaddy catches the drop then you’ll receive the domain right away. Since they only allow one back order per domain there is no post-drop auction like the other services. However, it seems like GoDaddy’s system rarely gets the drop. Unless the domain name you’re interested in is currently registered with GoDaddy then I would pass on using them.

Final Thoughts

The domain expiration process is still a bit like the wild-wild west. I think the basic problem is a conflict between the registrars and ICANN. The current system is tilted towards each individual registrar because they can create these exclusive partnerships with auction services. Undoubtedly they get a portion of the final sell price for each domain sent their way.

If ICANN was an effective organization they could set up a single waiting list or auction style system that would be better for everyone involved. Until they fix the current any-thing-goes system individual buyers are left open to potential fraud, lots of confusion, and an amazingly enormous waste of resources. Really, the best system that ICANN could figure out is one where people build big systems to continually ping the DNS system for the drop? Or where companies forge hidden back-room deals deciding where the domain goes? Why can’t the whole process be transparent and above-board where we as a society are able to better utilize the limited amount of domains.

There have been some proposals for fixing the current system [19]. However they seem to be tilted towards benefiting “domainers” instead of the common consumer. I wish ICANN would stand up and create a system that would be fair for end-users to understand and participate in. Until that time we’re stuck with this crazy system.

If you’re looking to try and catch an expiring domain name, good luck. I hope this article helped you navigate the waters and if it did please drop me a note letting me know.

References

[1] “How to snatch an expiring domain“, A great blog post by Mike Davidson back in 2005. This post was the inspiration for my post with updated information 5 years later.

[2] “How to Lose a Snap Names Domain Auction“, A funny blog post about the auction experience.

[3] “So best way to buy expiring domain?” A forum thread about purchasing expired domain names.

[4] “NSI Registry Registrar Protocol (RRP) Version 1.1.0“, The initial version of the RRP protocol

[5] “VeriSign Registry Registrar Protocol (RRP) Version 2.0.0“, A revised version of the RRP protocol.

[6] “Extensible Provisioning Protocol (EPP)“, The current version of the EPP protocol.

[7] “.COM Agreement Appendix 7“, The agreement between VeriSign and ICANN regarding the .com registry where VeriSign agrees to migrate to the EPP protocol.

[8] “Network Solutions: Domain Deletion Policy“, The agreement says that they will probably give you a grace period but by no means have too.

[9] “GoDaddy: What is your process for handling expired domain names?“, An FAQ about GoDaddy’s expiration process. Note that the agreement states all domains will be auction off by GoDaddy skipping the ICANN redemption period.

[10] “Name Cheap: Registration Agreement“, The agreement says that they will give you at least 27 days to renew the domain after expiration, but after that it’s up to them what they want to do.

[11] “1&1: What is a redemption period?“, As I read the FAQ from 1&1 they state that they will skip the registrar-hold status and immediately go to the redemption period (which imposes a penalty fee).

[12] “Tucows: What happens to domain names when they expire?“, When a domain expires Tucows will give you the full 45 day grace period to renew after which they make take the domain, auction it, or let it go into the redemption period before deleting.

[13] “Partnered Domain Names Understanding them“, a web page describing the partnerships between registrars and aftermarket auction services.

[14] “Which Drop Catch Service to Use for Expiring Domain Names“, An extensive list of registrar and aftermarket auction partnerships. I am not aware of how accurate this information is or when it was last updated.

[15] “SnapNames, Register.com Renew Deal“, Press release announcing the renewal of the exclusive partnership between SnapNames and Register.com.

[16] “Tucows Inks Expired Domains Deal with NameJet“, an article about the new partnership between Tucows and NameJet.

[17] “SnapNames sues former vice president for $33 million, alleging he rigged website auctions“, an article about the scandal surrounding SnapNames’s employee placing shill bids.

[18] “About NameJet“, this page lists the exclusive partnerships for NameJet. It’s the only one that lists them by name.

[19] “ICANN Tutorial: Changes in the expiry process“, Slides from a presentation given by the CEO of pool.com at a 2007 ICANN meeting on potential changes to the expiration process.

Way back in 1996 when I was in high school I first had the idea to register my name as a domain name. I looked it up at the time with a whois search and it was available. But at that time domains were much more expensive, it was on the order of $30 a year. I’m not sure if there were cheaper alternatives available at the time or not, but I wasn’t aware of any. That’s a fairly high price for a high school student with no job to register a domain name. I thought about it for a while and figured it would be worth it, even at the price. But by that time, just a few months later, someone else in Australia had picked up the domain. Presumably by another Scott Phillips…

Since then I’ve checked on the domain every so often. It’s never had a website hosted on it, the guy was just sitting on it. A few years later when I was out of high school and in college with a job I sent an email to the guy offering to buy it. I offered to buy it for $300 and he declined. I’d always imagined him laughing at me while writing the reply. I’ll never know.

When I got around to putting up my blog I needed a domain name and most of the Scott Phillips related names were already taken. Like the ones with dashes, or alternative extensions. However, there was one available on the relatively new “.name” registry. This registry is different because they sell third level names, so the registrar retains ownership of “phillips.name” and sells “scott.phillips.name”. This actually seemed like a cool idea it allows a lot more people to share the domain names separated by something that seemed natural.

The third level concept hasn’t caught on (yet?). You will have to search high and low for a respected registrar that will even let you register it, the big ones don’t support it Network Solutions, Namecheap, Go Daddy, etc. Also, I’ve been suspicious of using the domain for my primary email address because I fear that someday in the future I won’t be able to renew the registration on the domain. That’s why I hopped at the chance to get the .com version!




P.S. If you are another Scott Phillips reading this and wanted the domain name as well, unfortunately there’s no good way to share domain names. Sorry.

It turns out there’s a well known solution to the problem:

sudo rm -f /private/var/log/asl/*.asl

The speculation is the that a build up of log files causes the terminal app to continually slow down when opening a new prompt. After deleting the log files and opening a new terminal window the prompt appears immediately. I hadn’t noticed this behavior before Snow Leopard so it looks like apple forgot to clear something out on a regular basis or added a new check. In either case, it’s annoying.

References

[1] “OSX Terminal Slow To Launch”, http://www.proposedsolution.com/solutions/osx-terminal-slow-launch/

[2] “Speed up a slow Terminal by clearing log files”, http://osxdaily.com/2010/05/06/speed-up-a-slow-terminal-by-clearing-log-files/

[3] Apple Support Discussion on the problem, http://discussions.apple.com/thread.jspa?threadID=2178316

WordCamps are typically small 1-day conferences held all over the world focusing on WordPress related topics. For any given weekend there are likely several conferences being held. The big problem is that the camps almost always sellout before the first speaker is even scheduled. So you you either have to be ‘in the know’ about what are the good WordCamps or take your luck of the draw as to whether the camp will be right for you. I was afraid that the conference would be too flufy (i.e. topics like how to build a community, or just using WordPress) instead of technical topics. It turned out that most of the presentations were at least mildly technical with very few covering topics of just how to use WordPress. It seemed to be a direct match with the audience. I’d say that if you feel you’re a programmer you’re probably not going to get the most of out these conferences. Technical concepts are not covered in depth as you would expect from other communities.

With that said, I’m a newb to the WordPress community. I’ve built a few themes for my self (i.e. this blog) and a few others. So take what’s said here with a grain of salt from an outsider’s perspective.

Mood Cards

Presentation: Before the Famous Five Minute Install

Author: John Chandler

John’s presentation presented topics about what to do when building a new website — the stuff you should do before you start coding or designing. His presentation focused on techniques to illicit what the customer is looking for along with typical work flows. The basics of his topic were the straight forward water-fall design model. However one of the items he talked about was really interesting: Mood Cards. These are just simply a set of different web page styles to get the customer thinking about what they are looking for. The idea is that you either print these out and have your customer look through them in person, or send them a website with them all listed. With these the customer will have a starting point to say things like, a little bit of “this” and “that” – but I don’t like “this other one“. In short it helps start the conversation about designing a website.

Mood Cards online: http://moodcards.lyricalmedia.com/

Multiple Column Posts

Presentation: WordPress Beyond Blogging

Author: Bill Erickson

Bill’s presentation covered several topic areas focusing on how to use WordPress as a plain old CMS – more than just blogging. That seems to be the mantra going around the community recently. While he covered many topics, the one that I thought was most usefull was his tip to use the <h5> tag to separate out multiple columns. The idea is that instead of using short codes, or embedding HTML in the post, is to use the <h5> header tag to delineate when multiple columns should be displayed. A filter would look for these headings and add the appropriate HTML to separate out the content into multiple columns. The reason why this approch is better than short codes is that it works seemlessly with the WYSWYG editor. It’s also hard for the end-user editing their content to mess it up.

PODS is Crazy!

Presentation: The Sky’s the Limit – Migrating Static Sites using PODS

Author: Nick Batik

The next presentation was about a system called PODS. Nick’s presented this system as a very powerful plugin to build WordPress websites from sets of structurally defined data. The basic concept is that you’d import your data into a table, then configure WordPress to pull that information out to create pages/posts/comments/whatever using the data. He demonstrated a website he was in the process of building that pulled in static content from a book to publish it online.

The problem with PODS is that it has all the negatives of dynamic content (slow, complex, etc..) without any of the benefits of static content (fast, simple, easy to understand). The PODS plugin provides a horribly complex administrative interface that any end-user looking at it would give up after a few minutes. The user interface expects the end-user to provide embedded php code, know concepts about the loop, and other WordPress-isms. In short it’s basically un-usable. WordPress isn’t an end-all tool for every task. It’s a great platform to build your typically website with a great (probably the best) interface to allow end users to edit the content on their website. But PODS trys to extend WordPress beyond this use case and utterly fails. Perhaps there are some very small use-cases where PODS would be useful, such as a WordPress site solely administered by a php developer. But really, at that point you should step back and ask your self why? Why not just take that php developer and create some one-time scripts to convert the static content into static HTML. That would end up being much simpler to maintain over time, and probably easier that spending any amount of time figuring out the crazily complex PODS system.

Permalinks should must start with numbers

Presentation: Common WordPress Mistakes

Author: Jared Atchison

Jared’s presentation about common mistakes people make with WordPress included one that I hadn’t know about. Apparently when trying to find the correct page, post, or whatever for a particular URL it tries to optimize the search. If the first part of the path, the stuff before the first slash, is numeric then it will narrow down the search to just those posts that start with the number. If it’s something else then WordPress can’t do this optimization and has to check every type of object to see if it fits. This seems like an easy gotcha that could quite easily slow down your WordPress site. It also explains why I can’t create a page name that consists of nothing but a number.

If you stick with the permalink options provided you’ll be fine, they all start with a number.

Custom Post Types

Presentation: Custom Post Types

Author: Devin Price

Devin covered the new features from WordPress 3.0: Custom Post Types. These are a great step forward for the WordPress community and bring the platform up to par with Drupal, or Joomla. They have been needed for a while. Devin went into the concepts and explained several use-cases. I haven’t had a chance to use them but I’d like to add them to my other personal blog that I run.

Stephanie’s Static HTML Importer

Presentation: Content Importing

Author: Stephanie Leary

Stephanie’s presentation about importing content into wordpress was light on details. But it’s nice to know that there are lots of tools to aid in migrating content into WordPress. The one that she highlighted more than any other was her very own Static HTML Importer plugin. It definitely looked like a great plugin to use if you have a static HTML site that needs to be imported into WordPress. One interesting thing to note is that Stephanie works at A&M, in the University Writing Center. It is a bit weird to drive to Austin to hear a presentation by someone so close. In fact somewhere around half of the presenters and WordCamp Austin are or were living in the Bryan/College Station area in the last year. It seems to me that they really should have been WordCamp AggieLand.

Optimizing WordPress

Presentation: Optimizing WordPress

Author: Jason Cohen

Jason’s presentation was by far the best presentation that day. I wish some of the other presentations were a bit shorter so that Jason could have had more time. He covered caching and optimization topics, I suggest checking out this presentation when the video is posted. (I’ll post an update when that happens) Specifically he went through the options of the W3 Total Cache plugin, when to use them and when not to use them. One thing I learned is that the object cache is almost always a loss, you’re better off caching at the Database and HTML levels. It just dosn’t take PHP much time to recreate an object from the database so all your doing is wasting memory. He also cautioned about using disk caches if your disk speed is slow, like if you’re in a hosted cloud environment like RackSpace or Amazon’s EC2. In some circumstances you’re better of skipping the disk-based cache.

Another topic that I will highlight here is “Smush It” from Yahoo. The tool works as a plugin to Firefox and when run will take a webpage and compress the images to the smallest possible size without effecting the visual effect. Jason did a demo of using the tool on CNN‘s webpage and it was able to save 25% of the image bandwidth. That’s a pretty huge effect. I ran it on one of my websites and saw around a %10 savings. In either case, it’s a pretty nifty and easy to use tool.

Background

First a bit of background about HugeDomains.com. They are a relatively new outfit which started getting into the domain reselling business around 2006. The company behind HugeDomains.com is TurnCommerce.com operated by Andrew Reberry[4]. Andrew’s company is also affiliated with the domain registrar NameBright.com[5]. All of these companies are located in Denver, Colorado. The company TurnCommerce has an A+ rating with the BBB (as of November 2010). While you can disagree with their business model, they seem to be a reputable business. I can state that my interactions with the company were entirely professional, and once a deal was reached the transaction was processed quickly. We had the domain in our possession, transferred to my preferred registrar within 3 days.

The Value of a Name

Figuring out the value of domain names is tricky. There are lots of factors which can influence the value of any particular domain name. Here are a few general ones:

1) How short is the name? People like shorter names because they are easier to remember and quicker to type.

2) What is the potential business value? If the name is obviously linked to a product or service that people are looking for then its value is higher.

3) What extension is it? For whatever reason everyone want’s a “.com” domain name. All other extensions are worth a lot less.

4) Is the name brandable? Short or long, some names are easier for people to connect with a concept or business.

5) Does it contain weird characters? Things like dashes or numbers in strange places detract from the value of a domain.

6) Are there other alternatives? If people can choose something else for cheaper that also works, then the value is lower.

At the end of the day value is completely relative. How much a domain name is worth to me is different that what it is worth to someone else. HugeDomains.com and other resellers have to walk a fine line trying to figure out the market price for a particular domain. They try to figure out for someone who is interested in using this domain for it’s maximum potential (i.e. building a business on it) how much would that person be willing to pay for the domain. It’s a hard thing to do and the success of their business depends upon it.

Our Story

We were in the process of setting up a new website and needed a domain name. Unfortunately, it was already taken! HugeDomains.com already owned it and wanted $1,495 for it. In my opinion that’s an outrageously high sum of money for this particular domain name. Their website offers a one click buy button for the domain at the full asking price. Since we weren’t willing to do that, I sent an email inquiring about the domain to see if they even negotiate. Around a month later we received a reply from Christian Bosse, on behalf of hugedomains.com, he invited us to submit an offer for the domain.

At this point we needed to figure out how much we valued the domain. We were certainly not willing to spend the full asking price for the domain and after a long conversation about it we settled that our maximum amount we were willing to pay was $700, but that we’d make an an initial offer at $400. We received their reply to our initial offer very quickly, just a few hours later. The reply stated that HugeDomains won’t accept less than $500 for any domain that they sell. It then goes on to say that for domains in the $1500 range they typically accept offers in the $800 – $1000 range depending on “certain” factors. What those factors are, I have no idea. At they very end Christian adds one more wrinkle stating that our next offer would be “final”, what does that mean?

After receiving the reply we were unsure of what our next step should be:

As I saw it there were three options, 1) stick with the current $400 offer, 2) increase it to $500 the possible faux minimum, or 3) go straight to our $700 maximum. At this point we consult with several other co-workers having an engaging discussion on the best negotiation strategies to maximize returns. We had fun running through several scenarios and trying to predict what their response would be. However it boils down to the simple fact that we don’t know enough to predict their response or find the optimal strategy… it’s just a guess. In the end we decided that we would increase our bid to $500. Our plan was that they would either accept it right away or we would wait a few more months and try again at a slightly higher amount.

Our $500 offer was accepted right away. The domain transfer proceeded smoothly and everything worked just as expected. Happy ending, and we’ll have our website up soon.

Thoughts

I’ll never know if I should have stuck to my original offer of $400. I don’t know if they really have a $500 minimum policy or not. I am inclined to think I made a mistake and they would have taken the $400 offer. However I am glad that I didn’t jump immediately to my maximum bid!

Lastly, HugeDomains.com likely made a lot of money, even at the lower offer of $500. The prices that registrar’s pay for domains is fixed by ICANN at $6 for a 1 year registration [6]. However, we don’t know how much it costs them to acquire the domain initially nor do we know know the overhead of employing staff to handle negotiations, etc. It seems reasonable to assume that they picked up the registration for relatively cheap from someone else dropping their registration and I doubt their overhead costs for a small operation are that high. The business also operates on high volume with multiple hundreds of thousands of domains in their inventory. Factoring all these things in, it seems to me that their business model is highly profitable with extremely high markups.

References

[1] “Domain name speculation” on Wikepedia, http://en.wikipedia.org/wiki/Domain_name_speculation

[2] “HugeDomains.com = the next BuyDomains.com“,  http://www.domainstryker.com/huge-domains-andrew-reberry/

[3] “Dear HugeDomains.com: $995 is Too Much for this Blog Name – Ya Pirates.“, http://signalwriter.blogspot.com/2009/11/dear-hugedomainscom-995-is-too-much-for.html

[4] Andrew Reberry’s Linked in Profile, http://www.linkedin.com/in/andrewreberry

[5] “NameBright’s Terms and Conditions“, http://www.namebright.com/Terms.aspx

[6] “Revised VeriSign .com Registry Agreement: Appendix G“, http://www.icann.org/en/tlds/agreements/verisign/registry-agmt-appg-com-16apr01.htm

Updates

January 21st, 2011: The article was revised at the request of HugeDomains.com. I also took the opportunity to correct several grammatical problems and cleanup the article a bit.

Preparing Firefox profile...
dyld: Library not loaded: /usr/lib/libsqlite3.dylib
Referenced from: /System/Library/Frameworks/Security.framework/Versions/A/
   Security
Reason: Incompatible library version: Security requires version 9.0.0 or
   later, but libsqlite3.dylib provides version 1.0.0

What is Wrong?

The problem Firefox is complaining about is the version of libsqllite3.dylib found. Snow Leopard ships with a version of libsqllite in /usr/lib and Firefox also provides it’s own version of libsqllite3. Unfortunately there is a bug in Selenium 1.0.1 with how it calls Firefox that was patched in next version 1.0.2, however because of other bugs Tellerium is staying with 1.0.1 for now. You can read more about that decision in the mailing list thread:

http://www.mail-archive.com/tellurium-users@googlegroups.com/msg02149.html

The actual problem, at least from an outside view point, is dirt simple. When Selenium calls Firefox it sets up a set of environmental variables, one of which is:

DYLD_LIBRARY_PATH="null:/Applications/Firefox.app/Contents/MacOS"

As you can probably see there’s a problem with the path with the “null” that snuck in there. So somewhere in Selenium there’s some place where they forgot to check for a null value. When Firefox starts up it’s not able to find it’s local copy of libsqllight because of the invalid path arguments. But Firefox will happily guess a good set of paths if you forget to set the DYLD_LIBRARY_PATH, so a simple solution is to place a simple bash script in-between Selenium and Firefox that simple removes the corrupted path.

A Simple Solution

1) Create simple Selenium 2 firefox shell script which removes the corrupted path variable. Here’s mine:

#!/bin/bash

unset DYLD_LIBRARY_PATH;
/Applications/Firefox.app/Contents/MacOS/firefox-bin $@

2) Change Selenium to call your bash script directly instead of Firefox. Inside the connector configuration for your project you’ll have a line configuring your scripts to use Firefox, after the designation “*firefox” you can provide a path to where to find the browser. Change that path to the shell script that you created in step 1.

connectior {
     /** snip... **/
     browser = "*firefox /full/path/to/your/selenium2firefox.sh"

Conclusion

Beyond the basic hackish-nature of the working around the big downside is to using the approach is that Tellerium is not closing the old Firefox instances once it’s finished with them. That is at least a big improvement over the previous state of not able to run, so you can at least hobble along for a little while.

The import/checkout process normally this is a pain. However there are a few instances where it’s a really big pain such as Unix’s etc/ directory. You can’t just delete etc/ and recheck it out from version control or lots of stuff will break.  The other place I’ve found this usefull is for Xcode when starting new projects. Use the in-place import instead of Apple’s suggestion of creating two projects.

The process is quite simple. First create an empty directory is the repository, then checkout the empty directory into your existing location. Finally, run add the new files and then commit them into the repository.

svn mkdir https://your-svn-repo.com/new/directory/

svn checkout https//your-svn-repo.com/new/directory/  .

svn add *

svn commit –m “Initial in-place import of directory”

Our experiment upgrading to Grails 1.2.2 with Maven integration took a few days to work our way through the errors and gotchas.  In the end we were able to “successfully” upgrade our project – it compiled and passed all the tests. However it didn’t work that well. I’ve included my notes from the upgrade at the end of the post for anyone else who’s trying to do something similar. However in the end Grails + Maven integration sucked for a few reasons.

• Tests will fail if you do not run a mvn clean in-between. So you can’t run mvn test, fix something then, re-run mvn test again and expect it to work.
• Running the mvn package command required a larger amount of memory. I had to increase memory available via JAVA_TOOL_OPTIONS=-Xmx512m. This wasn’t required for our old Ivy-based build system nor is it required when using a pure grails build.
• The build is fragile, minor changes would break in unexpected ways. But the worst part is any errors encountered were always hidden behind java invocation errors masking the real error. This makes debugging the build much harder.
• In general the integration is very immature, maybe in the future this will improve. But at this state if I can’t recommend this to anyone.

In the end we decided that Maven integration just wasn’t worth it, and we needed something better. The next option we looked at was a pure grails-based build system. This option worked well, there we’re no errors or gotchas like the previous experiment. The best part is that it is able to pull dependencies from Maven without dealing with maven! It’s pretty simple, as described in the manual, just set your dependencies into grails-app/conf/BuildConfig.groovy. That’s it, you’re done. Run “grails compile” and you’re dependencies will be downloaded and the application compiled. This just took a an hour to figure it all out and show it working compared to the days it took getting the Maven integration barely working. In the end I highly recommend using a pure-grails based build, it is a 1000 times better than the traditional Maven integration.

Grails + Maven Notes

Here are the notes I took during the upgrade to Grails + Maven

1. Since no one has released Maven archetypes for Grails 1.2.2 we have to first upgrade using Maven to 1.2.0, then later upgrade to 1.2.2. Really how much different can the archetype be between these minor point versions?
2. Update you’re ~/.m2/settings.xml to include a reference to the Maven plugins for Grails.

   <plugingroups>
         <plugingroup>org.grails</pluginGroup>
   </pluginGroups>

3. Create a new pom.xml file for the project.

   mvn grails:create-pom –DgroupId=[groupId] -DartifactId=[artifactId]

   You’ll probably want to open the pom.xml file and add a few additional entries such as <name>, <description>, <url>, etc.

4. Next add your project’s dependencies into the pom.xml. This is obviously going to be very different for each project. But here are a few errors that I ran into for our specific project.

   Our project uses Xerces and Xalan for some XML processing, however these API’s conflicted with the Grails-based commands and their dependency on these libraries. Below is the error that I received (hopefully to help anyone searching for this error on Google). To fix the problem I needed to exclude our projects version of Xerces and Xalan and use the Grails version of these libraries.

   Embedded error: java.lang.reflect.InvocationTargetException
   loader constraint violation: when resolving overridden method "org.apache.
   xerces.jaxp.SAXParserImpl.getXMLReader()Lorg/xml/sax/XMLReader;" the
   class loader (instance of org/codehaus/groovy/grails/cli/support/
   GrailsRootLoader) of the current class, org/apache/xerces/jaxp/
   SAXParserImpl, and its superclass loader (instance of <bootloader>),
   have different Class objects for the type org/xml/sax/XMLReader
   used in the signature
   

   The second gotcha was our dependency on dumbster:dumbster:1.6 which records dependencies on both javax.activation:1.0.2 and javax.mail:mail:1.3.2 both of which have entries in the Maven central repository but no artifacts exists with the entries. It’s weird, you can get the pom.xml files but not the actual jar files. I fixed this by adding in dependencies for the new versions of both libraries.

5. Run the maven upgrade command. In theory you should just run “mvn grails:exec -Dcommand="upgrade"”, but theory doesn’t always work. There’s some bug with running the upgrade command, it will fail with the following error:

   Embedded error: java.lang.reflect.InvocationTargetException
   /Users/scott/Development/Eclipse/Workspaces3.5/DSpace/trunk/null/src/war not found.
   

   To get around this you will need to install Grails, and run the upgrade command directly.

   GRAILS_HOME=/path/to/temporary/grails/home
   cd <project/dir>
   $GRAILS_HOME/bin/grails upgrade
   unset GRAILS_HOME
   

6. Check you plugins. You can use “mvn grails:list-plugins” to get your current list. Then use “mvn grails:uninstall-plugin –DpluginName=<name>” and “mvn grails:install-plugin –DpluginName=”<name> <version>”“.
7. Remove the old build system’s such as any build.xml files left over.
8. Change the pom.xml files to reference the latest version of Grails. You should also use the same steps from step 5 to run the Grails upgrade command.

Alexey Maslov, James Creel, Adam Mikeal, Scott Phillips, John Leggett, Mark McFarland. Adding OAI­-ORE Support to Repository Platforms. Journal of Digital Information, Volume 11, Number 1. Available at http://journals.tdl.org/jodi/article/view/749

The goals of our project were to create a mechanism where we could run complete functional tests. Functional tests evaluate the entire system as the end user would use it, so think of it as opening a web browser and evaluating the output – but completely automated. They test everything all together. Ideal it would be better to test each component individual, but this is in practical for DSpace for two reasons 1) DSpace is highly integrated and nearly impossible to separate from the database and file systems, 2) Creating unit test for all of DSpace is very time consuming it is simpler to write a few functional tests that cover a wide set of features over the whole application. It gets you to a point where you can reliably verify the software quicker. If you’re working on unit tests for DSpace please do not let this stand in your way.

The main concept is to script the install of a test DSpace, with a full configuration and setup. Then we start DSpace in an embedded webserver and then run through several scenarios just as a normal user would. This tests the whole application, using a database, a file system, and a full build. The ant script where you normally run “ant fresh_install” has a new target “ant test”. You pass it a few parameters such as what database to use. The script will then run through a fresh install of DSpace into a local /test directory, setup some communities and collections, and import some basic items. Then JUnit-based tests are run against the embedded webserver using HtmlUnit to simplify verifying the HTML output.

Here is how to run it. After compiling using a “mvn package”, cd into target/dspace-*-build.dir/ directory. Then run “ant test” you may need to pass it some parameters as listed below. Each parameter has a default so if you configure you’re database connections the same way then it can be as simple as running “ant test” without any parameters.

 -Dtest.db.driver="org.postgresql.Driver"
 -Dtest.db.url="jdbc:postgresql://localhost:5432/dspacetest"
 -Dtest.db.username="dspacetest"
 -Dtest.db.password="dspacetest"
 -Dtest.dspace.dir=”./test/"
 -Dtest.config=”./test/config/dspace.cfg"

We’ve used this approach rather successfully for two of our DSpace-based projects here at TDL: an ETD submission system called vireo, and a learning object repository. These projects haven’t moved to 1.6 yet, but I do have a patch available for DSpace 1.5.2. Most of the test cases we’ve created so far are specific to the project we’re working on. However the patch includes 4 manakin tests, which are really just an example of how tests work within this framework.

Download the patch: DSpace-1.5.2-FunctionalTest-V2.patch

The question is, is this something that the DSpace community would like? You can follow the discussion of this topic on the dspace-devel mailing list.

Update 4/8/2010: The original patch was missing a class, that has been corrected.

Versions:

Eclipse: 3.5
Tomcat: 6.x
DSpace 1.5.x or 1.6.x

1) Download Eclipse

Obtain a fresh version of Eclipse from the official download site, from which there are several variants. You can choose either “Eclipse for Java EE Developers” or “Eclipse for Java Developers”. Once you’ve selected the variant, choose the distribution that is appropriate for you (Mac, Windows, Linux, and 32bit vs 64bit). Once downloaded, unpack the software where you would like it to be installed, however it’s best if you do not try start Eclipse until after step #2.

Download Eclipse, either Java for EE Developers or just Java for Developers.

2) Install the Sysdeo Tomcat plugin

The Sysdeo Tomcat plugin allows you to start, stop, and restart Tomcat from within Eclipse. The plugin also allows you to debug a running web application by setting break points, stepping through the code line by line, and examining memory locations through the standard Eclipse debugger. Download the latest version of the Sysdeo Tomcat plugin and un-compress the software into Eclipse’s plugin directory. Next start Eclipse for the first and the plugin will be recognized. If you have previously started Eclipse then it will not immediately recognize the new plugin. In this case you need to start Eclipse with the “-clean” option from the command line.

http://www.eclipsetotale.com/tomcatPlugin.html

3) Install the SVN & Maven plugins

Help -> Install New Software

Fortunately for the other plugins the process is easier than the Tomcat plugin, you will be able to use the built-in updating mechanism. There are two more plugins that you need: Subversive and M2Eclipse. Subversive enables you to checkout the DSpace source code and keep up-to-date with the latest changes to the platform from DSpace’s SVN Repository. Maven‘s M2Eciplse plugin is used to support DSpace’s new Maven-based build system. Begin by selecting “Help -> Install New Software…” from the main menu.

Near the top of the Install New Software dialog box there is a button labeled “Add…”, this lets you add a new site the list of places where Eclipse will look to get software from. We need to add a new site for M2Eclipse’s plugin. Click the “Add…” button and enter the information blow into the Add Site dialog box.:

Name: M2Eclipse

Location: http://m2eclipse.sonatype.org/update

Add new M2Eclipse site.

Click “OK” to add the new site. You will be returned to the Install New Software dialog box, change select field labeled “Work With” to “–All Available Sites–”. This will bring up a big list of all the available software you can install from all knows sources, including the one you just added.

Select the "--All Available Sites--" option.

From the large list of Eclipse plugins and other components select the three components listed below for SVN and Maven:

• Collaboration
  • Subversive SVN Team Provider (Incubation)
• Maven Integration
  • Maven Embedder
  • Maven Integration for Eclipse (Required)

Once the plugins are selected, click the “Next >” button and follow the dialogs to install the plugins. After they are installed you will be prompted to restart Eclipse which you should agree too. When Eclipse restarts there should be a dialog box waiting for you to select which SVN implementation to use. You can either choose to use JavaHL which uses your local system implementation or SVNKit which is a pure Java implementation. If you clicked the window away, or it did not appear, then switch to the “SVN Repository Exploring” perspective and the dialog box will appear.

Select which type of SVN implementation to us, either JavaHL to use your OS's local binaries or SVN Kit for a pure Java implementation.

4) Check out the DSpace source code

The next step is to check out the DSpace source code from the DSpace SVN Repository. If the “Welcome” tab is displayed when you start eclipse, click that away to return to the workbench. In the Project Explorer panel, left hand side, select “New -> Project…” from the context menu.

Project Explorer -> New -> Project...

This will bring up a “Select a wizard” dialog box, select entry the “Project from SVN” wizard and click the next button.You will then be asked to configure an SVN Repository to check out code from. In the URL field put DSpace’s repository URL and leave all other fields at their defaults. Then click next.

URL: http://scm.dspace.org/svn/repo/

Enter the Root DSpace SVN Repository

You may be asked to accept a certificate from the repository, if so, accept it.

The next step will ask you to select a specific version of DSpace to checkout. You can choose a specific taged version such as 1.5.1 or 1.5.2. These tags will never change, they have already been released. Alternatively you can choose to check out a branch or mainline trunk these versions will continue to change over time as new features and bug fixes are added.  Click the “Browse…” button to select the version of DSpace you want to work with. When you are done, click the “OK” button followed by the “Finished” button.

Select which version of DSpace to check out, either a tag, branch or the main line trunk.

The final step will ask you where to place DSpace within your Eclipse workspace. Select the last option “Check out as a project with the name specified:” and enter any name you choose.

5) Configure Maven

After checking out DSpace’s source code the next step is to compile. DSpace uses Maven as its build system but before you can start to compile you’ll need to configure Maven. Locate the project you just checked out in the Project Explorer on the left hand side of you Eclipse workbench. Right clicking on the project will bring up a context menu. There will be a Maven submenu with several options, you need to enable dependency management and nested modules. After ensuring those two options are turned on use Maven to update the project’s configuration. Between each of these steps Eclipse may need to rebuild its workspace, however after it is finished you should not see any compilation errors.

• Maven -> Enable Dependency Management
• Maven -> Enable Nested Modules
• Maven -> Update Project Configuration

Configure maven and update the project's configuration.

Next you need to create a “Run Configuration” to compile DSpace. From Eclipse’s top tool bar select “the green arrow -> Run Configurations…” as shown below:

The Green Arrow -> Run Configurations...

Next the “Run Configuration” dialog box will appear. Locate “Maven” on the list, and right click to select the “New” option from the context menu.

Create a new Maven-based run configuration.

This will change the right hand side of the dialog box displaying the details for a maven command. Fill out the following parameters as shown below in the picture.

Name: MVN Package
Base Directory: ${project_loc}
Goals: package

Next add a new parameter “dspace.config” that points to the full path to you dspace.cfg.

Detailed view of the completed run configuration

When you are finished click the “Run” button to compile DSpace. I suggest you also create another Run Configuration for the Maven “clean” goal.

6) Configure Tomcat

The last component needing configuration is the Tomcat Sysdeo Plugin. This plugin enables you to start and stop Tomcat from within Eclipse and run web applications in the debugger. However the plugin needs to know where your Tomcat is installed and where to run the web application. First, start with general Tomcat configuration. Select the main Eclipse preferences, “Eclipse -> Preferences” from the main menu (under windows this may be located in the File menu).

Select the appropriate Tomcat version and path.

Inside the “Preferences” dialog box expand the “Tomcat” option from the left hand hierarchy. At this screen select the correct version of Tomcat that you are using, and enter the full path to the Tomcat directory. Next select the “Tomcat Manager App” from the left hand hierarchy and enter your Tomcat username and password. If you have not set up a Tomcat manager account then use the “Add user to tomcat-users.xml” button to create one. When you are finished click “OK”.

Provide a username and password to authenticate with Tomcat's manager application.

The next step is to configure the DSpace project to use Tomcat. In the main menu select the project’s properties by “Project -> properties”, and select “Tomcat” from the left hand hierarchy. Enter the following fields and click “OK”.

Is a Tomcat Project: Check

Context Name: dspace

Subdirectory: dspace-xmlui/dspace-xmlui-webapp/
              target/dspace-xmlui-webapp-1.5.2/

You need to change the sub-directory to point to a complete Webapplication ARchive (WAR) file or directory. This directory will change if you want to check out the JSPUI or the XMLUI, and it also changes slighly for each different version of DSpace. Below are some examples you can use to extrapolate where which directory to use.

DSpace JSPUI 1.5.2: dspace-jspui/dspace-jspui-webapp/target/dspace-jspui-webapp-1.5.2/

DSpace XMLUI 1.5 branch: dspace-xmlui/dspace-xmlui-webapp/target/dspace-xmlui-webapp-1.5.2-SNAPSHOT/

Declare the project as a Tomcat Project, set a context name, and provide the path to the web-application.

9) Deploy the web application

Congratulations your done! To test that everything work start Tomcat, then use the Update and Reload commands to publish DSpace’s web-application into Tomcat. Use you’re web browser to check that DSpace is running.

Webtest is the more established project with its first release in 2007. Webtest is provided by Canoo as a free open source plugin to Grails. The tests are run via an Ant script that packages together the test cases and runs through each one.

Functional-test is a relatively new project with its first release in early 2009. Functional test is basically the same as webtests however instead of using the Ant-based scripts everything is based on Junit and implemented in plain old java or groovy.

We’ve chosen to proceed with using the Junit-based plugin for our HTTP integration tests. Our primary reason in deciding to use Junit is that it integrates well with our other unit-based tests which also use Junit instead of using disparate frameworks for the various types of tests. While this plugin may be newer it’s traffic on the mailing lists is growing and substantial.

Django Applications vs Projects

Django provides tools and concepts to break up your websites into very small components that can be re-used between sites. The first and most basic distinction that needs to be understood is Django’s difference between applications and projects.

A Django Application is a single self-contained set of related features that depends upon the Django framework. By convention, a Django application typically will contain: admin.py, models.py, urls.py, views.py, and other files. These applications should contain no configuration that needs to be changed between installations.

A Django Project is a collection of applications that share a single database configuration, work together in a URL space, and share a common set of application configurations. Projects may contain multiple applications or just one. Django requires that projects have at least three files: manage.py, settings.py, and urls.py. I believe it is best to consider Django Projects like Apache’s httpd.conf.

Application Layout

django_application/
    |
    |---------- module_name/
    |              |
    |              |---------- models.py
    |              |
    |              |---------- templates/
    |              |
    |              |---------- urls.py
    |              |
    |              |---------- views.py
    |
    |---------- site_media/
    |              |
    |              |---------- module_name/
    |
    |---------- docs/

The application is where all the code that accomplishes whatever tasks the application requires will reside. It is best to split applications into the smallest possible units to increase the possibility that your application can be reused in another project. As outlined above, there should be three key directories:

1. a python module,
2. site_media, and
3. a directory for any relevant documentation.

The python module will contain all the python source files for your application, this can be easily created using the django-admin.py command ‘startapp‘. Each application should have a set of default templates that reside in a special directory named ‘templates/‘ inside the python module. The default template loaders know to search inside this directory for each applications. The next major directory, ‘site_media/‘, should contain all the static content that is to be served directly from the web server. In order to support the most flexible options during application deployment by local convention, all the files should reside in a sub-directory named the same as the python module. This will make it easier for the web server to separate site media from multiple applications effectively. When building your application make sure that all references to static content take the form of ‘/site_media/module_name/*‘, this way multiple application’s static content can co-exist in the same URL name space.

Project Layout

django_project/
    |
    |---------- manage.py
    |
    |---------- settings.py
    |
    |---------- urls.py

Django project should contain only a minimal amount of source code, instead focusing on configuration for the applications. The default layout of projects can be created quickly using the django-admin.py command. These files should be in some sort of version control but they will contain passwords and settings that change between environment that make it difficult to share. This is the same problem any system administrator will face with other configuration files. Locally we use a separate part of our version control repository just for configuration that has limited access set aside just for these types of configuration files.

How to Deploy

Following the patterns outlined here ultimately provides a set of flexible deployment options of your applications and site, avoiding situations where passwords and database connection parameters reside in your source code repository. Each application may have specialized installation instructions. However, the steps outlined below will work for most applications. This assumes that python, django, your database, and apache are already installed.

1) Install the Django Applications

First, check out the desired applications from the code repository and install them into your local python installation. There are three methods you can use:

A) Symlink the python module_name/ directory into python’s site-packages.
(Probably best when actively developing)

B) Run the install script to compile a python egg, and install in site-packages.
(Probably best in a production environment)

C) Modify the pythonpath environmental variable to include the python module.

2) Create a Django project

Use the django-admin.py program to quickly create a new Django project for your site. Using the command will create a default project with all the basic configuration files awaiting your editing.

$ django-admin.py startproject 

3) Customize settings.py

The settings file controls a lot of how your website will work. Below are the four major areas of settings that need to be configured for a particular installation.

A) Database & URL settings

Configure your project to use a particular database. This includes all the DATABASE_* settings for your particular environment. You also need to configure the URL parameters for media.

B) Installed applications

Configure the INSTALLED_APPS array to include the applications desired for your Django project. Because these applications are installed directly as a python site wide package, all you need to do is include the module_name of each application in the array.

C) Application specific settings

Each application will typically contain a set of parameters that need to be configured before it will operate properly. Consult each application’s documentation for specific instructions.

4) Customize urls.py

Django’s urls.py file controls what particular URLs are mapped to specific functions. At the site level you need to map your installed applications into the website’s URL namespace. Some applications may require a particular namespace, and others may require multiple mappings. For a simple basic configuration, add a line for each application as shown below, where URL-NAMESPACE is replaced with the url under which this application should be installed and MODULE-NAME is the application to install.

(r'^URL-NAMESPACE/', include('MODULE-NAME.urls')),

For example, if you want the application ‘myapp‘ to be used for all URLs that begin with ‘myapp‘, then use the following line:

(r'^myapp/', include('myapp.urls')),

As another example, if you want one application (ex: myapp) to be installed at the root of the URL namespace, use the following line:

(r'.', include('myapp.urls')),

5) Setup Apache directives

There is a basic Apache configuration needed for Django to operate properly. Beyond this basic configuration listed below, an additional step is required to enable across to each application’s static content. If your site is using the default templates provided by the application(s), then you need to map each application’s site_media directory so that it is accessible over the web.

Just for reference, this is the basic Django configuration:

SetHandler python-program
PythonHandler django.core.handlers.modpython
SetEnv DJANGO_SETTINGS_MODULE mysite.settings
PythonPath "['/opt/auth/sites'] + sys.path"

Then for each application, map the corresponding location using Apache’s aliases and locations. One possible configuration is:

Alias /site_media/app1 /full/path/to/app1/site_meda/app1

Alias /site_media/app2 /full/path/to/app2/site_media/app2

SetHandler None

Resources:

These ideas are not all my own. Here are the resources I used while creating this guide:

• James Bennett. Practical Django Projects. Apress (June 23, 2008). ISBN 978-1590599969.
  Available from Amazon, Apress, or see a preview on Google Books.

• “Django tips: laying out an application”,
  http://www.b-list.org/weblog/2006/sep/10/django-tips-laying-out-application/

• A forum thread where someone asks the basic question, how should I layout my Django application?
  http://stackoverflow.com/questions/44135/project-design-fs-layout-for-large-django-projects

• The semi-official Django Do and Don’t list.
  http://code.djangoproject.com/wiki/DosAndDontsForApplicationWriters

Citation:

Mikeal, A., Creel, J., Maslov, A., Phillips, S., Leggett, J., and McFarland, M. 2009. Large-scale ETD repositories: a case study of a digital library application. In Proceedings of the 2009 Joint international Conference on Digital Libraries (Austin, TX, USA, June 15 – 19, 2009). JCDL ’09. ACM, New York, NY, 135-144. DOI= http://doi.acm.org/10.1145/1555400.1555423

Identity: The URL a blog is hosted at is extremely important. First, it is one of the first things new readers will notice about your blog, and second it must persist. I have chosen to go with the relatively new top-level domain: “.name“. I find the domain interesting. It is a new breed of domains where registrars are able to sell 3rd level domains, i.e. I own scott.phillips.name, but I do not own phillips.name as the generic surnames are reserved. The URL is a clean representation of what the site is, this is the blog of Scott Phillips – a person. As for the second property, persistence, because I own the domain it will persist for as long as this blog is relevant. I will have the option to migrate the blog between technologies and across services. Most of the free or near free blogging services support independently owned domains along with self hosting, however TDL’s service does not support these types of URLs.

Customization: Writing a blog is a significant investment of time that should not be wasted on a poorly built site. I want to be able to have a unique look-and-feel that represents my blog, and have control over the version availability of plugins. This level of customization is impossible for most blogging services to provide because the many support issues which can be created. I want the ability to do custom programming, or more likely tweaking code to my taste. The only option that supports this is self publishing a blog, really no shared service can provide this.

Credibility: The TDL blogging service hosted at the blogs.tdl.org domain, along with all the other services offered by TDL, gives a blog the instant credibility that the blog is scholarly in purpose and raises it above the numerous blogs. For others this may be a more important factor, but I feel this is not enough of an incentive to overcome the other two factors.

Microsoft announced their intention to engage with the repository community at last year’s OR08 conference in Southhampton. The OR09 workshop featured the tools that Microsoft has developed in the intervening year starting with new authoring tools in Word, publishing via SWORD, a new peer-reviewed journaling service, and the new .NET based repository. It’s clear to see that Microsoft will have a huge impact on scholarly publishing in the future. We, as the repository community, will need to adjust our services to ensure they work smoothly within the Microsoft ecosystem. After the jump read about the three big features that were presented.

Authoring Addin:

Let’s start with Microsoft’s Word suite. There are several “addins” Microsoft is in the process of creating for Word. “Addins” are plugins that can be downloaded from Microsoft providing extra functionality. The most interesting of these new addins is “Authoring Addin” for Word 2007 (windows only) still in Beta. This pushes metadata creation down into the very first steps the author takes when creating their document.

The tool is based upon Word templates, so you as a journal publisher would create a Word-based template for your journal that gives the visual style of your journal, the required sections, and what metadata fields are required or optional. The creator downloads this template and begins their work. When they put the title into the document’s body it is automatically extracted because they are using the template. Next, after the creator is finished and ready to submit their article to a journal, they will be stopped until they fill in the required metadata fields. Within Word, a pane slides out querying the user for additional metadata. Once the required metadata fields are present, the user can continue their submission via SWORD. The user doesn’t need to know URLs or metadata schemas, all that technology is hidden behind the template. From the creator’s point of view, if they start with the correct template then they just use Word as normal and click the publish button at the end – Very simple. However, all this hinges on starting with the correct template.

Electronic Journal Service:

Next let’s continue with Microsoft’s new journaling service. The first thing to note is that this will be a service and not software. It will be hosted by Microsoft.  The service handles the entire peer review process for an on-line academic journal up to, but not including, publication. The service, in alpha testing at the moment, appears to be geared for both non-profit and commercial publishers. The service does not place any constraints on the publishing of the journal. Material is to be exported to whatever system the publisher wants to use. This integrates nicely with the author tools and SWORD compatibility described earlier.

The best method to submit content into the journaling service is via SWORD from within Word, although a standard web-based form submission is also available. The service offers a similar work flow to Open Journal System’s model. However, it doesn’t bewilder the user with a massive array of options. One of the interesting points is that if articles are submitted as Word documents then all the workflow can be accomplished within Word’s native GUI. As an editor you review articles by downloading them, marking them up, filling in or correcting metadata all within Word before uploading your updated version. Once an article has passed through the peer review process and is ready to be published then the article will be pushed to the publisher via SWORD, or if that is not available, then FTP.

This is interesting to see this use of ‘double SWORD.’ It is something we have been thinking about for a workflow system we’re at the very beginning stages of here in Texas: accept SWORD as input and then when finished with the review workflow, automatically push the content into the repository via SWORD.

Zentity, .NET repository platform

Finally let’s cover Microsoft’s new repository platform: Zentity. This has received lots of hype within the repository community. Zentity is a typical web-based application built upon the Microsoft stack. Its data model underneath is a classic triple store with some basic scholarly object relationships defined. Zentity implements a wide array of interoperability protocols: RSS, OAI-PMH, OAI-ORE, SWORD (and Atom Pub) just to start. Microsoft obviously is not using Zentity to trap customers into a Microsoft platform.

After looking at the software it’s only feature that distinguishes it from other repository platforms is just that its on the Microsoft stack. In fact, it feels like not much development has gone into what is currently implemented; it feels like the current features were just the low lying fruit that are easy to implement because it is on the Microsoft stack. I’m hard pressed to see any reason why someone would switch to Zentity or, if they don’t have a repository, choose Zentity over another platform.

Commentary, Where is the IR use-case?

The big thing that I noticed to be absent from Microsoft’s plan is the use-case for the Institutional Repository in their ecosystem. Microsoft Word integrates into journal publishing via SWORD and the electronic journaling service but where does Zentity, or any other repository fit into that? If a faculty member downloads a journal’s Word template and then submits it into the journal’s submission system then published on the journal’s website how does it get captured into the institutional repository?

The template-based design lacks a critical feature in my opinion, ‘A me too’ feature. Something that streamlines a faculty member’s submission to both the journal and the institutional repository with just one click. Without something that fills this need, faculty members will be forced to create one document and submit it to the journal of their choice via SWORD, then using another template from the IR create a second document to submit to the institutional repository. This is anything but one-click submission!

A common question for someone just starting to develop with DSpace is how do others set up their development environment. Often times this isn’t documented anywhere, but a lot of time goes into researching the best way to set things up. Today I co-taught a class on customizing DSpace for TDL. One of the hand outs I created for the class is a simple how to setup DSpace, Eclipse, and Tomcat together for easier development. This is certainly not the only way to set up these tools, but it is the method most developers in TDL choose.

One thing to note is the use of Sysdeo’s Tomcat plugin vs. Eclipse’s WTP plugin for integration with Tomcat. I played around with the WTP tools that come standard with Eclipse and found them to be too buggy to rely on. I hope that future versions of Eclipse will iron out the kinks in WTP tools, but for now I’m going to stick with Sysdeo.

Versions:

• Eclipse 3.4.x
• Tomcat 5.x or 6.x
• DSpace 1.5.x

Howto: setup Eclipse, Tomcat, and DSpace

This last Wednesday after getting back from vacation I co-taught the “DSpace Customization” class with Steve Williams from UT. I think the class was a great success, here are the slides we used for the class.

DSpace Customization Slides

Obviously this isn’t going to work for another department, so I created a simple one page hand out (print in duplex) to give someone who needs to create an import. I think this is still a good resource, so even if I wrote it over a year ago I wanted to publish it on my blog.

DSpace Batch Import Format