Documentum 16.3 delayed until Feb 2018

If you access the new support page, there is a section where you can find roadmaps for the products, and if you check the Documentum related ones, you’ll see the target date for ths new version listing new features. Without going into much detail (As I’m not sure how public this info is), these caught my atention:

  1. Ongoing exposure of D2 APIs through REST (goodbye DFS?)
  2. Native support for S3 object storage protocols to alleviate storage costs (goodbye OnDemand, hello AWS?)
  3. New REST Services (see #1, no mention of DFS anywhere to be found)
  4. Webtop will get a 16.3 version (yeah!)

 

Support adventures II (REST Services edition)

I wasn’t expecting this week to be so “productive” 🙂

I had the chance to engage again with Documentum support, which is always an adventure 😉

We’re currently porting a framework from DFS to REST services. A couple of days ago, one of my colleagues had a problem and ask me if I knew why a query through the REST services was failing.

The query is using DATETOSTRING to retrieve a time field and returning a column with the attribute name using the alias, so, if a user requests expiration_date we execute a query like:

select DATETOSTRING(expiration_date,’dd/mm/yyyy’) as expiration_date from…

which returns the requested attribute in the desired (configurable) format. Easy, right? Well, the problem was that using r_creation_date or r_modify_date was throwing an exception:

 {    “status”: 400,
“code”: “E_INPUT_ILLEGAL_ARGUMENTS”,
“message”: “There are illegal arguments provided.”,
“details”: “Invalid format: \”12/04/2013\” is malformed at \”/04/2013\””}

We also observed that changing the alias to something else than r_creation_date/r_modify_date would work (but it was breaking our use case).

This was weird, because I was quite sure that those kind of queries worked previously, so I checked against the Content Server:

Connected to Documentum Server running Release 7.1.0210.0328  Linux64.Oracle
1> select DATETOSTRING(r_creation_date,’dd/mm/yyyy’) as r_creation_date from dm_document enable (return_top 1);
2> go
r_creation_date
—————
12/04/2013
(1 row affected)

As I though, it was working just fine. I suspected it had something to do with the custom date format, as EMC/OpenText tends to forget that not everyone uses ANSI or the american date format, so I decided to open a SR.

After some exchange of emails, we were told that both r_creation_date and r_modify_date where reserved words and that it was expected to fail. What???

After asking support to either fill this as a product limitation or fixing it, I decided to take a look at the code myself.

As I suspected, the query works just fine, and results are returned to the Query controller. The problem here is with the way REST generates the response. If you run the query with a different alias to see the result page you’ll get this:

"entries": [  {
"id": "http://127.0.0.1:8080/dctm-rest/repositories/repo.json?dql=select%20DATETOSTRING_LOCAL(r_creation_date,%27dd/mm/yyyy%27)%20as%20rr_creation_date%20from%20dm_document%20enable%20(return_top%201)&index=0",
"title": "12/04/2013",
"updated": "2017-06-16T11:04:38.284+00:00",
"published": "2017-06-16T11:04:38.284+00:00",
"content": {
"json-root": "query-result",
"definition": "http://127.0.0.1:8080/dctm-rest/repositories/repo/types/dm_document.json",
"properties": {
"rr_creation_date": "12/04/2013"
}}}]

Do you see those fields before the content element? That’s where everything breaks, why? Because of this:

public Date entryUpdated() {
Date updated = null;
Object modifyDate = ((QueryResultItem)getDataInternal())
  .getMandatoryAttribute("r_modify_date");
if (modifyDate == null) {
updated = new Date();
} else if ((modifyDate instanceof Date)) {
updated = (Date)modifyDate;
} else {
updated = DateFormatter.parse(modifyDate.toString());
}
return updated;
}

public Date entryPublished()
{
Date published = null;
QueryResultItem queryResultItem=getDataInternal();
Object modifyDate = ((QueryResultItem)getDataInternal())
  .getMandatoryAttribute("r_creation_date");
if (modifyDate == null) {
published = new Date();
} else if ((modifyDate instanceof Date)) {
published = (Date)modifyDate;
} else {
published = DateFormatter.parse(modifyDate.toString());
}
return published;
}

As you can see (besides the obvious copy/paste from one method to another changing the name of one variable), the standard view looks in the results from the query for a column with those names, and if it founds a column matching that name, tries to parse it with a forced format (which of course, it’s miserably failing with our custom date format).

This, in my opinion, is a bug, because the behaviour of the query functionality it is inconsistent between the Documentum stack, in fact, this query only fails with REST services, so I’ll keep pushing the SR to be treated as a bug and fixed by the talented team, and not as a “product limitation” or a “feature request”.

And if you face the same problem, and it is still not fixed, you have three options:

  • Keep waiting for a fix
  • Extend com.emc.documentum.rest.view.impl.QueryResultItemView and handle the IllegalArgumentException that throws DateFormatter.parse
  • Extend the controller adding a custom view for the results and removing/overriding the updated/published fields.

Filename in Documentum REST Services

If you have used Document REST Services you most likely have realized that downloading any content from an object return a “document”/”response” + dot + file extension.

You can check William Zhou’s answer here:

This had been discussed in the initial implementation but hadn’t been implemented since Content-Disposition response header is neither mandatory nor handled by all HTTP clients. Besides, it has some overhead to map the format/mime to a filename extension. But I think it has values helping for the download experience. It is appreciated if you can file a feature request CR so that we can discuss this with the product manager.

I’ve alredy raised a SR to support in order to get OpenText to consider adding this “feature”.

However, if you can’t/don’t want to wait, just “extend” com.emc.documentum.rest.controller.ContentMediaController adding the following line to the getContent method just before returning the response:

headers.setContentDispositionFormData("attachment", (String)co.getAttributeByName("object_name"));

and you’re good to go.

 

Updated roadmap, webinar and FAQ from OpenText Documentum

If you registered for the webinar OpenText held a few weeks ago (if you didn’t, maybe you can check Andrey’s post on the subject), you should have received an invitation to some site from OT with a FAQ about the Documentum stack. I’m not going to paste it here, just in case it’s not public, but IMHO, the highlights are:

  • Content Server: Aligment with ECD. No new features but trying to move to a microservices architechture. It probably means CS won’t evolve anymore.
  • Webtop: Several points on this. Will keep being supported and updated to support latest CS.
  • Rest: Every new product based on this. No mention about DFS (I hope it’s dead, unless they decide to update with libraries from this decade)
  • xCP: It looks like xCP 1.x support will be over by the end of 2019.
  • Support: Still through EMC’s site until the end of the year.

Customizing/Extending REST services (II)

This is an improved version of the “hack” posted in the previous post (Customizing/Extending REST services)

  • Decompile com.emc.documentum.rest.controller.QueryController
  • Look for method executeQuery and modify the following line as you wish:

Preconditions.checkArgument(DQLChecker.isSafeQuery(dql), MessageBundle.INSTANCE.get(“E_UNSAFE_QUERY_DQL”, new Object[] { dql }));

You can use something like:

Preconditions.checkArgument(true, MessageBundle.INSTANCE.get(“E_UNSAFE_QUERY_DQL”, new Object[] { dql }));

if you want everyone to be able to run write queries or you can allow these queries to be executed only by some user(s) with something like:

if (RepositoryContextHolder.getLoginName().equalsIgnoreCase(“dmadmin”)){
//skip DQLChecker check for select statements
}

  • Finally place the new class in dctm-rest/WEB-INF/classes/com/emc/documentum/rest/controller.

CMIS vs REST comparison with code examples (Custom WebDAV server for Documentum)

Although My Documentum for Desktop is a nice product when you want a deeper integration with Windows, if you only need WebDAV access for browsing and creating documents it can be too much for the users. So I decided to develop a custom WebDAV server using a 1.x version of milton.io’s WebDAV library (newer 2.x free version doesn’t seem to work well with Office documents).

This library provides a WebDAV Servlet that you can customize for providing your own objects as WebDAV elements. I used CMIS (with Apache Chemistry) to query the repository and provide a WebDAV access to a Documentum repository. I wasn’t happy with the performance so I tried with the REST API as well.

The following is a comparison of features and code between those two implementations:

  CMIS REST
Compatibility Multiple CMS Documentum only
Memory footprint * Big Small
Ease of implementation ** Very easy Easy
Speed/Performance Fast Slightly faster (noticeable difference)

* Memory footprint: Although I’m still not sure of the source of the problem, CMIS consumes every Documentum session specified in dfc.properties. So if you have 100 max concurrent sessions, it’s going to use every single one. This is less noticeable when using a WebDAV client (session usage is between 7-20 sessions) but when using Windows integrated WebDAV client it goes crazy after 2nd level of folders. Not sure if it is due to windows querying every single object displayed or something related to CMIS specification or the implementation in emc-cmis.war

** Ease of implementation: When using CMIS with Apache Chemistry, you can code every single operation with a couple of lines of code.

Using the REST API will require a little bit more coding (I used org.json library to parse the responses):

Create document

  • CMIS:
Folder tempFolder=(Folder)session.getObjectByPath(path);
Map<String, Object> properties = new HashMap<String, Object>();
properties.put(PropertyIds.OBJECT_TYPE_ID, "dm_document");
properties.put(PropertyIds.NAME, newName);  

ContentStream contentStream = new ContentStreamImpl(newName,
          BigInteger.valueOf(length), contentType, in);
Document newDoc = tempFolder.createDocument(properties, contentStream, VersioningState.MAJOR);  

return newDoc.getId();
  • REST:
HttpPost httpPost=new HttpPost(pathId.replace("/objects/","/folders/")
          .replace(".json","/objects.json"));
httpPost.addHeader("Authorization",authorizationString);  

HttpClient client = new DefaultHttpClient();  

String json = "{\"properties\" : {\"object_name\" : \"" + newName +
          "\", \"r_object_type\" :\"dm_document\"}}";  

File temp = File.createTempFile(newName,".tmp");
byte[] data = IOUtils.toByteArray(in);
FileUtils.writeByteArrayToFile(temp, data);  

MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("data", new StringBody (json,
          "application/vnd.emc.documentum",Charset.forName("UTF-8")));
reqEntity.addPart("content", new FileBody(temp, newName, contentType, null));  

httpPost.setEntity(reqEntity);
HttpResponse response = client.execute(httpPost);
temp.delete();  

String result=getResponseAsString(response.getEntity().getContent());
JSONObject obj= new JSONObject(result);
JSONArray jsonArray = (JSONArray) obj.getJSONArray("links");  

return (String)jsonArray.getJSONObject(0).get("href");

Create Folder:

  • CMIS:
    Folder tempFolder=(Folder)session.getObjectByPath(path);
    Map<String, Object> properties = new HashMap<String, Object>();
    properties.put(PropertyIds.OBJECT_TYPE_ID, "cmis:folder");
    properties.put(PropertyIds.NAME, newName);  

    Folder newFolder=tempFolder.createFolder(properties);  

    return newFolder.getId();
  • REST:
    HttpPost httpPost=new HttpPost(pathId.replace("/objects/","/folders/")
                   .replace(".json","/objects.json"));
    httpPost.addHeader("Authorization",authorizationString);  

    HttpClient client = new DefaultHttpClient();  

    String json = "{\"properties\" : {\"object_name\" : \"" + newName +
              "\", \"r_object_type\" :\"dm_folder\"}}";  

    MultipartEntity reqEntity = new MultipartEntity();
    reqEntity.addPart("data", new StringBody (json,
              "application/vnd.emc.documentum",Charset.forName("UTF-8")));  

    httpPost.setEntity(reqEntity);
    HttpResponse response = client.execute(httpPost);  

    String result=getResponseAsString(response.getEntity().getContent());
    JSONObject obj= new JSONObject(result);
    JSONArray jsonArray = (JSONArray) obj.getJSONArray("links");  

    return (String)jsonArray.getJSONObject(0).get("href");

Version object:

  • CMIS:
Document newDoc = (Document) session.getObject(session.createObjectId(objectId));  

ObjectId newVersionId = newDoc.checkOut();
Document newVersion=(Document)session.getObject(newVersionId);
ContentStream cstream = new ContentStreamImpl(newDoc.getContentStream().getFileName(),
               BigInteger.valueOf(length), newDoc.getContentStream().getMimeType(), in);
newVersionId=newVersion.checkIn(true, null, cstream, null);  

return newVersionId.getId();

REST:

HttpGet httpGet=new HttpGet(pathId);
httpGet.addHeader("Authorization",authorizationString);  

HttpClient client = new DefaultHttpClient();  

//get lock-checkout url
HttpResponse response = client.execute(httpGet);  

String result=getResponseAsString(response.getEntity().getContent());
JSONObject obj=new JSONObject(result);
JSONArray jsonArray = (JSONArray) obj.getJSONArray("links");  

String lockUrl=null;
for (int i=0; i<jsonArray.length(); i++){
     if (jsonArray.getJSONObject(i).get("rel").toString().endsWith("checkout")){
         lockUrl=jsonArray.getJSONObject(i).get("href").toString();
     }
}  

//lock-checkout object
HttpPut httpPut=new HttpPut(lockUrl);
httpPut.addHeader("Authorization",authorizationString);  

response = client.execute(httpPut);
result=getResponseAsString(response.getEntity().getContent());
obj=new JSONObject(result);
jsonArray = (JSONArray) obj.getJSONArray("links");  

String objName=obj.getJSONObject("properties").get("object_name").toString();
String checkinUrl=null;  

//get checkin url
for (int i=0; i<jsonArray.length(); i++){
     if (jsonArray.getJSONObject(i).get("rel").toString().endsWith("checkin-next-major")){
         checkinUrl=jsonArray.getJSONObject(i).get("href").toString();
     }
}  

String json = "{\"properties\" : {}}";  

File temp = File.createTempFile(objName,".tmp");
byte[] data = IOUtils.toByteArray(in);
FileUtils.writeByteArrayToFile(temp, data);  

//cheking new version
HttpPost httpPost=new HttpPost(checkinUrl);
httpPost.addHeader("Authorization",authorizationString);  

MultipartEntity reqEntity = new MultipartEntity();
reqEntity.addPart("data", new StringBody (json,
             "application/vnd.emc.documentum",Charset.forName("UTF-8")));
reqEntity.addPart("content", new FileBody( temp , objName, contentType, null));  

httpPost.setEntity(reqEntity);  

response = client.execute(httpPost);
result=getResponseAsString(response.getEntity().getContent());
obj=new JSONObject(result);
jsonArray = (JSONArray) obj.getJSONArray("links");  

return (String)jsonArray.getJSONObject(0).get("href");

Consuming RestFul services with GSON

If you want to consume the RestFul services you can check Wei Zhou’s posts in the documents section of the documentum developer network (Documentum) or check EMC RestFul Services — Clients JAVA by astone to see an example of how to consume the JSON directly from JAVA.

However, while developing the Android app I found that it was way easier to go the old Java-objects way, and you can do this by using GSON (or any other similar library).

First, you’ll need to create the POJOs/Beans for the classes; as we don’t have those (thanks EMC…) you’ll have to code them by yourself from the JSON responses (or use something like http://www.jsonschema2pojo.org/ to autogenerate them).

Once you have those (and the Gson libraries in your project) it is quite straightforward:

Reading a JSON response as an object:

DefaultHttpClient httpClient = new DefaultHttpClient();  
      
//documentum user and password  
String authorizationString = "Basic " + Base64.encodeToString((strUser + ":" + strPass).getBytes(), Base64.NO_WRAP);  
      
//request the page, here we get the main repository page  
HttpGet getRequest = new HttpGet(currentServer +"/repositories");  
//don't forget the credentials  
getRequest.setHeader("Authorization", authorizationString);  
//get the response  
HttpResponse getResponse = httpClient.execute(getRequest);  
HttpEntity getResponseEntity = getResponse.getEntity();  
Reader reader = new InputStreamReader(getResponseEntity.getContent());  
      
//use gson to get the object  
Gson gson=new Gson();  
RepositoryList repoList = (RepositoryList)gson.fromJson(reader, RepositoryList.class);  

Modify some property by sending a POST request:

DefaultHttpClient httpClient = new DefaultHttpClient();  
  
//documentum user and password  
String authorizationString = "Basic " + Base64.encodeToString((strUser + ":" + strPass).getBytes(), Base64.NO_WRAP);  
  
//setting up the POST request  
HttpPost postRequest = new HttpPost(currentServer+"/repositories/"+currentRepo+"/objects/"+strId);  
//don't forget the credentials  
postRequest.setHeader("Authorization",authorizationString);  
  
//Create the object with the property you want to modify  
RepositoryObject ro=new RepositoryObject();  
Attribute props=new Attribute();  
  
props.setSubject("new Subject");  
ro.setProperties(props);  
  
//set the header to the POST request  
postRequest.setHeader("Content-Type", "application/vnd.emc.documentum+json; charset=utf-8");  
//convert the object to json using gson  
Gson gson=new Gson();  
postRequest.setEntity(new StringEntity(gson.toJson(ro)));  
  
//do the POST  
HttpResponse postResponse = httpClient.execute(postRequest);

Note: As you can see, I’m not using service discovery although that would be the right way to do it, however, this was faster and it works as a how-to