CMIS REST interface slow performance with multiple repositories

When using the REST interface provided by CMIS, if you have multiple repositories on your docbroker, you may have noticed a slight delay in the response to your queries. I came across this issue in an environment with multiple repositories (+10), where additionally, every dmadmin user has different password, so not only we had this delay on the responses, we also had a log full of authentication errors.

If the delay was slighty concerning, the errors thrown on the log pointed to something more serious (and most likely a known issue in D2).

With a little bit of debugging and decompiling, we found out the culprit on opencmis-dctm-connector-XXX.jar, in the com.emc.documentum.fs.cmis.impl.cmisoperations.impl.GetRepositoryInfosOperationHandler class. This (surprise surprise) is the same situation that we saw with D2. Check the getRepositoryInfoList method:

private List getRepositoryInfoList(CallContext context, GetRepositoryInfosOperationInputBean inputBean, List repositoryEntries)
   {
     List repositoryInfoList = new ArrayList();
     for (CmisRepositoryEntryType repositoryEntry : repositoryEntries)
     {
       String repositoryId = repositoryEntry.getRepositoryId();
       if (CmisAuthenticationManager.INSTANCE.authenticate(context, repositoryId)) {
         repositoryInfoList.add(getRepositoryInfo(context, inputBean, repositoryEntry));
       }
     }
     return repositoryInfoList;
   }

Yes, they’ve done it again. The code tries to authenticate with the provided credentials in evey single repository known to the docbroker. Luckily for us, this method has the requested repository (i.e.: http//cmis…/resources/RepositoryName/…) in the context variable, so you can just do a comparison there and “magically” improve the perfomance (or wait until Opentext releases a “fix”).

Note that this happens in every version I could test from 16.4 to 20.4 😦

CMIS vs REST comparison with code examples (Custom WebDAV server for Documentum)

Although My Documentum for Desktop is a nice product when you want a deeper integration with Windows, if you only need WebDAV access for browsing and creating documents it can be too much for the users. So I decided to develop a custom WebDAV server using a 1.x version of milton.io’s WebDAV library (newer 2.x free version doesn’t seem to work well with Office documents).

This library provides a WebDAV Servlet that you can customize for providing your own objects as WebDAV elements. I used CMIS (with Apache Chemistry) to query the repository and provide a WebDAV access to a Documentum repository. I wasn’t happy with the performance so I tried with the REST API as well.

The following is a comparison of features and code between those two implementations:

  CMIS REST
Compatibility Multiple CMS Documentum only
Memory footprint * Big Small
Ease of implementation ** Very easy Easy
Speed/Performance Fast Slightly faster (noticeable difference)

* Memory footprint: Although I’m still not sure of the source of the problem, CMIS consumes every Documentum session specified in dfc.properties. So if you have 100 max concurrent sessions, it’s going to use every single one. This is less noticeable when using a WebDAV client (session usage is between 7-20 sessions) but when using Windows integrated WebDAV client it goes crazy after 2nd level of folders. Not sure if it is due to windows querying every single object displayed or something related to CMIS specification or the implementation in emc-cmis.war

** Ease of implementation: When using CMIS with Apache Chemistry, you can code every single operation with a couple of lines of code.

Using the REST API will require a little bit more coding (I used org.json library to parse the responses):

Create document

  • CMIS:
Folder tempFolder=(Folder)session.getObjectByPath(path);
Map<String, Object> properties = new HashMap<String, Object>();
properties.put(PropertyIds.OBJECT_TYPE_ID, "dm_document");
properties.put(PropertyIds.NAME, newName);  

ContentStream contentStream = new ContentStreamImpl(newName,
          BigInteger.valueOf(length), contentType, in);
Document newDoc = tempFolder.createDocument(properties, contentStream, VersioningState.MAJOR);  

return newDoc.getId();
  • REST:
HttpPost httpPost=new HttpPost(pathId.replace("/objects/","/folders/")
          .replace(".json","/objects.json"));
httpPost.addHeader("Authorization",authorizationString);  

HttpClient client = new DefaultHttpClient();  

String json = "{\"properties\" : {\"object_name\" : \"" + newName +
          "\", \"r_object_type\" :\"dm_document\"}}";  

File temp = File.createTempFile(newName,".tmp");
byte[] data = IOUtils.toByteArray(in);
FileUtils.writeByteArrayToFile(temp, data);  

MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("data", new StringBody (json,
          "application/vnd.emc.documentum",Charset.forName("UTF-8")));
reqEntity.addPart("content", new FileBody(temp, newName, contentType, null));  

httpPost.setEntity(reqEntity);
HttpResponse response = client.execute(httpPost);
temp.delete();  

String result=getResponseAsString(response.getEntity().getContent());
JSONObject obj= new JSONObject(result);
JSONArray jsonArray = (JSONArray) obj.getJSONArray("links");  

return (String)jsonArray.getJSONObject(0).get("href");

Create Folder:

  • CMIS:
    Folder tempFolder=(Folder)session.getObjectByPath(path);
    Map<String, Object> properties = new HashMap<String, Object>();
    properties.put(PropertyIds.OBJECT_TYPE_ID, "cmis:folder");
    properties.put(PropertyIds.NAME, newName);  

    Folder newFolder=tempFolder.createFolder(properties);  

    return newFolder.getId();
  • REST:
    HttpPost httpPost=new HttpPost(pathId.replace("/objects/","/folders/")
                   .replace(".json","/objects.json"));
    httpPost.addHeader("Authorization",authorizationString);  

    HttpClient client = new DefaultHttpClient();  

    String json = "{\"properties\" : {\"object_name\" : \"" + newName +
              "\", \"r_object_type\" :\"dm_folder\"}}";  

    MultipartEntity reqEntity = new MultipartEntity();
    reqEntity.addPart("data", new StringBody (json,
              "application/vnd.emc.documentum",Charset.forName("UTF-8")));  

    httpPost.setEntity(reqEntity);
    HttpResponse response = client.execute(httpPost);  

    String result=getResponseAsString(response.getEntity().getContent());
    JSONObject obj= new JSONObject(result);
    JSONArray jsonArray = (JSONArray) obj.getJSONArray("links");  

    return (String)jsonArray.getJSONObject(0).get("href");

Version object:

  • CMIS:
Document newDoc = (Document) session.getObject(session.createObjectId(objectId));  

ObjectId newVersionId = newDoc.checkOut();
Document newVersion=(Document)session.getObject(newVersionId);
ContentStream cstream = new ContentStreamImpl(newDoc.getContentStream().getFileName(),
               BigInteger.valueOf(length), newDoc.getContentStream().getMimeType(), in);
newVersionId=newVersion.checkIn(true, null, cstream, null);  

return newVersionId.getId();

REST:

HttpGet httpGet=new HttpGet(pathId);
httpGet.addHeader("Authorization",authorizationString);  

HttpClient client = new DefaultHttpClient();  

//get lock-checkout url
HttpResponse response = client.execute(httpGet);  

String result=getResponseAsString(response.getEntity().getContent());
JSONObject obj=new JSONObject(result);
JSONArray jsonArray = (JSONArray) obj.getJSONArray("links");  

String lockUrl=null;
for (int i=0; i<jsonArray.length(); i++){
     if (jsonArray.getJSONObject(i).get("rel").toString().endsWith("checkout")){
         lockUrl=jsonArray.getJSONObject(i).get("href").toString();
     }
}  

//lock-checkout object
HttpPut httpPut=new HttpPut(lockUrl);
httpPut.addHeader("Authorization",authorizationString);  

response = client.execute(httpPut);
result=getResponseAsString(response.getEntity().getContent());
obj=new JSONObject(result);
jsonArray = (JSONArray) obj.getJSONArray("links");  

String objName=obj.getJSONObject("properties").get("object_name").toString();
String checkinUrl=null;  

//get checkin url
for (int i=0; i<jsonArray.length(); i++){
     if (jsonArray.getJSONObject(i).get("rel").toString().endsWith("checkin-next-major")){
         checkinUrl=jsonArray.getJSONObject(i).get("href").toString();
     }
}  

String json = "{\"properties\" : {}}";  

File temp = File.createTempFile(objName,".tmp");
byte[] data = IOUtils.toByteArray(in);
FileUtils.writeByteArrayToFile(temp, data);  

//cheking new version
HttpPost httpPost=new HttpPost(checkinUrl);
httpPost.addHeader("Authorization",authorizationString);  

MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("data", new StringBody (json,
             "application/vnd.emc.documentum",Charset.forName("UTF-8")));
reqEntity.addPart("content", new FileBody( temp , objName, contentType, null));  

httpPost.setEntity(reqEntity);  

response = client.execute(httpPost);
result=getResponseAsString(response.getEntity().getContent());
obj=new JSONObject(result);
jsonArray = (JSONArray) obj.getJSONArray("links");  

return (String)jsonArray.getJSONObject(0).get("href");

Testing D2 development and REST services

After the summer, I’ve found myself with some spare time between projects that I’ve used to review some EMC products:

  • D2 4.1

Over the years, we’ve developed a webtop based solution that includes several custom developed components like import from external source, TBO’s/SBO’s used to name documents/folders, etc.

So after attending several D2 demos, we decided to do exactly the same with D2 (or at least get as close as possible) as a testing project.

Custom naming/default object creation was the easiest part, as the autonaming and autolink features are functional enough to cover our needs.

Import from external source looked like it was going to be tough, as we couldn’t touch the default import, and the development documentation is… lacking. So after a little investigation thorugh the available documentation and the community, we decided to go with the external widget approach and it was quite easy actually: Take our custom applet from webtop, put it on a jsp and communicate with the D2 import component sending the files… no problem at all

Some other functionality that involved modifying default D2 behavior required to developed plugins, but after reviewing the API and the available examples we decided not even trying because currently is a waste of time trying to figure which method/libs you need.

  • Good: external widgtes are extremely easy to develop, and there is enough documentation to be able to communicate those widgets with D2 without killing yourself in the process.
  • Bad: plugins are useless as there’s no reliable documentation available, and it seems the API could change in future versions, making useless old plugins.

To sum up, I think D2 4.1 is more a beta/unfinished product (which it really is, as EMC is doing everything from scratch to made it compatible with browsers other than IE, and currently is missing features that were on previous versions). I’ve heard that D2 4.2 is a more mature product and that effors have been made regarding development documentation. If EMC wants to “kill” wdk application with D2 and xCP2 they better give us enough tools to do our customizations, as by now we all should realize that the idea of “configure only” is cool, but is far from our daily work…

  • REST Services:

I’ve been developeing different solutions with DFS (extra layers to the services, custom clients both in JAVA and .NET, bulk load applications, etc) since version 6.0. Afterwards I migrated everything that could be done with the CMIS standard to CMIS with Apache Chemistry; so it was clear the next step was trying the new REST services.

Before the summer I started developing an Android application with the REST preview just as a proof of concept (and as a way to learn how android applications work), and it worked quite well. Last week I’ve been updating the application using the REST 7.0 services (finally we have a working query service!!!) and cleaning/improving the android application that has now become an administrator tool that lets you check repository status, browse the repository, open files, check job status, etc.

I hope to keep improving the application and I hope EMC keeps the REST services, as there is way easier to develop with these REST services than using the monstruosity that DFS are. However, if CMIS meets your requirements, I still go that way, as it is a standard…