Documentum REST documentation

So today someone asked us if it was possible to link/unlink folders via REST api. We were somehow surprised because this is a quite straightforward API, so we took a look at the swagger documentation where we found (surprise!) a “parentLink” resource:

This seems simple enough, right? Well, while trying this on postman first question popped up: How do we specify the target? Umm… weird enough… anyway let’s try the example… great, it doesn’t work because the body is wrong… Let’s check the documentation… great, it doesn’t even mention the controllers because these are obviously “documented” on swagger… So what do we do? Well, let’s decompile the controller:

@RequestMapping(method = {RequestMethod.POST}, produces = {"application/vnd.emc.documentum+json", "application/vnd.emc.documentum+xml", "application/json", "application/xml"})
  @ResponseBody
  @ResponseStatus(HttpStatus.CREATED)
  @ResourceViewBinding({FolderLinkView.class})
  public FolderLink link(@PathVariable("repositoryName") String repositoryName, @PathVariable("objectId") String childId, @RequestBody FolderLink folderLink, @RequestUri UriInfo uriInfo) throws DfException {
    validateTargetControllerAccessible(ParentFolderLinkController.class);
    String parentId = folderLink.getObjectId();
    if (parentId == null)
      throw new RestClientErrorException("E_OBJECT_ID_NOT_FOUND", null, HttpStatus.BAD_REQUEST, null); 
    ResourceReferenceValidator.validate(folderLink.getHref(), parentId, RESOURCE_NAMES_TO_VALIDATE);
    this.folderLinksManager.link(childId, parentId);
    FolderLink newFolderLink = new FolderLink(parentId, childId, false);
    Map<String, Object> otherParams = new HashMap<>();
    otherParams.put("link_to_parent", Boolean.valueOf(true));
    otherParams.put("post_from_collection", Boolean.valueOf(true));
    return (FolderLink)getRenderedObject(repositoryName, (Linkable)newFolderLink, true, uriInfo, otherParams);
  }

The code seems simple enough, as the POST expects a FolderLink object, but this fails with the example. Why? Let’s check the FolderLink class:

@SerializableType(value = "folder-link", jsonWriteRootAsField = false, fieldVisibility = SerializableType.FieldVisibility.NONE, fieldOrder = {"href", "child-id", "parent-id", "links"}, xmlNS = "http://identifiers.emc.com/vocab/documentum", xmlNSPrefix = "dm")
public class FolderLink extends AbstractLinkable {
  @SerializableField(xmlAsAttribute = true)
  private String href;
  
  @SerializableField(value = "child-id", xmlAsAttribute = true)
  private String childId;
  
  @SerializableField(value = "parent-id", xmlAsAttribute = true)
  private String parentId;
  
  private String objectId;
  
  public FolderLink() {
    this.href = null;
    this.childId = null;
    this.parentId = null;
    this.objectId = null;
  }

Great, no properties element. So that’s why the example miserably fails. So let’s add a child-id and a parent-id attributes and this will work, right? Wrong. Another error stating that the source id can’t be null. What the heck? Let’s take a look at the controller’s line where it calls “getObjectId”:

public String getObjectId() {
    if (Strings.isNullOrEmpty(this.objectId) && 
      !Strings.isNullOrEmpty(this.href))
      this.objectId = IdExtracter.extract(this.href); 
    return this.objectId;
  }

So… The swagger example prompts you to use “properties” as an element in the POST body, while, if you remotely want this to work, you have to provide a href element with the object you want to use as parent:

<?xml version="1.0" encoding="UTF-8"?>
<folder-link>
    <href>https://server/dctm-rest/repositories/test_repo/folders/0c0180aa80001107</href>
</folder-link>

As you can see, almost exactly as the swagger example (and very logical approach too, because providing parentid was too complex I guess…)

Remote DAR install

A usual challenge when trying to automate Documentum operations is how to streamline the installation of dar files. These are done via a huge application (composer, basically a customized eclipse) that teams usually mount into some container / server to run these intalls.

However, there’s a simpler way to get this to work: by using a REST endpoint.

1. Create a method to run the dar intall from the content server by running a command line script:

java -Ddfc.keystore.file=$DOCUMENTUM/config/dfc.keystore -Ddar=$1.dar -Dlogpath=/tmp/darinstaller.log -Ddocbase=$2 -Duser=dmadmin -Ddomain= -Dpassword=dmadmin -cp $DM_HOME/install/composer/ComposerHeadless/startup.jar org.eclipse.core.launcher.Main -data $DM_HOME/install/composer/workspace -application org.eclipse.ant.core.antRunner -buildfile $DM_HOME/install/composer/deploy.xml

Note that this is an example where we’re willingly ignoring the user/password authentication as this will be delegated to the REST call.

2. Create a REST extention point to run this. This a simple example of the controller class:

public ContentfulObject createObject(@PathVariable("repositoryName") final String repositoryName,  @RequestBody final InstallDarInfo createObject,  @TypedParam final SingleParam param,  @RequestUri final UriInfo uriInfo)
        throws Exception {

    createObject.addAttribute(new Attribute("object_name",createObject.getDar()));
    createObject.setType("dm_document");

    ContentfulObject result = sysObjectManager.createSysObjectUnderParentFolder(createObject, "/Temp/installDAR", true, param.getAttributeView());

    Map<String, Object> params = Collections.singletonMap(ViewParams.POST_FROM_COLLECTION, (Object) true);  

    runInstallDARMethod(repositoryName, result.getId(), (String)result.getAttributeByName("object_name"),"/Temp/installDAR") ;
    return (ContentfulObject) getRenderedObject(repositoryName, (ContentfulObject)result, param.isLinks(), uriInfo, params);
}

private void runInstallDARMethod(String repository, String objectId, String fileName, String folderPath) throws DfException {

    String dqlMethod="execute do_method with method='m_InstallDAR', arguments='"+ objectId + " " + fileName + " " + repository + " " + folderPath + "', launch_async=true, run_as_server=true";
    this.queryEngine.execute(QueryResultItem.class, dqlMethod+";", QueryType.QUERY, 0, 100);
}

As you can see, this simply takes the file attached to the REST call, stores it on a temporary folder on the repository, and then calls the method to run this.

With this, you can also handle something that, if you’ve played with Documentum cloud images, you might have already realized that OT engineers do not know: The additional artifacts that come with DAR files (install parameters, locales, referenced dar files, etc. The usual stuff “nobody” uses in the real world). Also, you can process several files (ie: zip file containing everything needed to install), you can store the output log, return the log, use different build files depending on your needs, etc.

However, this still presents a challenge: you need to deploy this on DCTM-REST and create a method to run the script that needs to be placed on the CS.

So, is there anything else we can do? Yes 😀

From a couple versions back (20.x?) Documentum has included a JMS servlet to run DAR installs (InstallDarServlet). This is a rather “simple” class that basically receives a couple of parameters (repository, user, login ticket and an object_id from a dar file existing in the repository) and it will run a simple dar install. This servlets presents “great room for improvement”, so you can create a class with the same name and package, copy the code (so you don’t break whatever OT is using this for) and then add a handler for a multipart REST message which does everything we’ve discussed before and then replace this class on the CS. By doing this you will get:

  1. Simplest deployment for deploying DAR files automatically (just replacing one class and restarting JMS, as the servlet is already present on web.xml)
  2. You really don’t need to store anything on the repository, this can be run synchronously (be aware of long running DAR installations) and return the whole log, or you can store everything in the repository as audit trail.
  3. You can handle install parameters, locales, referenced dar files, etc. (which again, seems something that OT engineering have never heard of, who really uses locales? everyone loves systems on English :D)
  4. You can control the access to this servlet via user/password, by allowing only certaing IPs to call it and using trusted login to install DAR files, etc

Documentum REST dm_job object on DQL queries

If you have used dctm-rest 21.x, you might have come across this error when doing something like select * from dm_job:

E_INPUT_ILLEGAL_ARGUMENTS: There are illegal arguments provided. Failed to instantiate class class com.emc.documentum.rest.view.impl.QueryResultItemView with arguments..

You might wonder if you did something wrong when deploying dctm-rest or if there’s something missing on your environment. Well, the answer is no. If you change the DQL to something like:

select *,r_object_id as method_id from dm_job

Magic! It works 🙂

Well, the “issue” is on the com.emc.documentum.rest.view.impl.JobView class, where someone at some point decided that it was a good idea to require a non-existing attribute (method_id) in dm_job view class:

protected void addMethodsLink()
{
String methodId = (String)((JobObject)getDataInternal()).getMandatoryAttribute("method_id");
makeLinkIf(!methodId.equals("0000000000000000"), CoreLinkRelation.METHOD.rel(), idUriBuilder("method", methodId).build());
}

Also, if you want a more “permanent” solution you can override this method with something like:

protected void addMethodsLink()
{
String methodId = (String)((JobObject)getDataInternal()).getMandatoryAttribute("method_id");
if (methodId!=null){
makeLinkIf(!methodId.equals("0000000000000000"), CoreLinkRelation.METHOD.rel(), idUriBuilder("method", methodId).build());
}
}

And you’ll be good to go.

CMIS REST interface slow performance with multiple repositories

When using the REST interface provided by CMIS, if you have multiple repositories on your docbroker, you may have noticed a slight delay in the response to your queries. I came across this issue in an environment with multiple repositories (+10), where additionally, every dmadmin user has different password, so not only we had this delay on the responses, we also had a log full of authentication errors.

If the delay was slighty concerning, the errors thrown on the log pointed to something more serious (and most likely a known issue in D2).

With a little bit of debugging and decompiling, we found out the culprit on opencmis-dctm-connector-XXX.jar, in the com.emc.documentum.fs.cmis.impl.cmisoperations.impl.GetRepositoryInfosOperationHandler class. This (surprise surprise) is the same situation that we saw with D2. Check the getRepositoryInfoList method:

private List getRepositoryInfoList(CallContext context, GetRepositoryInfosOperationInputBean inputBean, List repositoryEntries)
   {
     List repositoryInfoList = new ArrayList();
     for (CmisRepositoryEntryType repositoryEntry : repositoryEntries)
     {
       String repositoryId = repositoryEntry.getRepositoryId();
       if (CmisAuthenticationManager.INSTANCE.authenticate(context, repositoryId)) {
         repositoryInfoList.add(getRepositoryInfo(context, inputBean, repositoryEntry));
       }
     }
     return repositoryInfoList;
   }

Yes, they’ve done it again. The code tries to authenticate with the provided credentials in evey single repository known to the docbroker. Luckily for us, this method has the requested repository (i.e.: http//cmis…/resources/RepositoryName/…) in the context variable, so you can just do a comparison there and “magically” improve the perfomance (or wait until Opentext releases a “fix”).

Note that this happens in every version I could test from 16.4 to 20.4 😦

Java 11 Documentum REST Client examples

On Java 11, HttpClient was added to Java (no more need to use Apache’s HttpClient). and this presents some changes to the usual way of consuming Documentum REST services. The main “issue” is that there is no MultiPart class provided to easily handle multipart content (create a document), so we need to do it manually. I’ve upload a project to GitHub with basic examples of using Documentum REST services with Java 11: DctmRest11

OTEW 2018 fun

I’m not attending the event being held in Toronto, but I found through twitter this nice url:

http://hol-host05.eastus.cloudapp.azure.com:81/d2-unity-web/ui/app.html -> This is the new D2 UI (and yes, you can use the you-know-which-default-user(s) to log in and check it by yourself) deployed on Azure (which is weird, considering Opentext has its own cloud…)

But, the really funny thing here, are these urls:

http://hol-host05.eastus.cloudapp.azure.com:81/da -> da 7.3 (but with CS 16.4/SQL Server)

http://hol-host05.eastus.cloudapp.azure.com:81/D2 -> hello old D2 vulnerabilities 🙂

http://hol-host05.eastus.cloudapp.azure.com:81/d2-unity-web/repositories -> and you can log in with you-know-which-default-user(s), and you have a nice DQL tool provided by REST services 🙂

Documentum 16.3 delayed until Feb 2018

If you access the new support page, there is a section where you can find roadmaps for the products, and if you check the Documentum related ones, you’ll see the target date for ths new version listing new features. Without going into much detail (As I’m not sure how public this info is), these caught my atention:

  1. Ongoing exposure of D2 APIs through REST (goodbye DFS?)
  2. Native support for S3 object storage protocols to alleviate storage costs (goodbye OnDemand, hello AWS?)
  3. New REST Services (see #1, no mention of DFS anywhere to be found)
  4. Webtop will get a 16.3 version (yeah!)

 

Support adventures II (REST Services edition)

I wasn’t expecting this week to be so “productive” 🙂

I had the chance to engage again with Documentum support, which is always an adventure 😉

We’re currently porting a framework from DFS to REST services. A couple of days ago, one of my colleagues had a problem and ask me if I knew why a query through the REST services was failing.

The query is using DATETOSTRING to retrieve a time field and returning a column with the attribute name using the alias, so, if a user requests expiration_date we execute a query like:

select DATETOSTRING(expiration_date,’dd/mm/yyyy’) as expiration_date from…

which returns the requested attribute in the desired (configurable) format. Easy, right? Well, the problem was that using r_creation_date or r_modify_date was throwing an exception:

 {    “status”: 400,
“code”: “E_INPUT_ILLEGAL_ARGUMENTS”,
“message”: “There are illegal arguments provided.”,
“details”: “Invalid format: \”12/04/2013\” is malformed at \”/04/2013\””}

We also observed that changing the alias to something else than r_creation_date/r_modify_date would work (but it was breaking our use case).

This was weird, because I was quite sure that those kind of queries worked previously, so I checked against the Content Server:

Connected to Documentum Server running Release 7.1.0210.0328  Linux64.Oracle
1> select DATETOSTRING(r_creation_date,’dd/mm/yyyy’) as r_creation_date from dm_document enable (return_top 1);
2> go
r_creation_date
—————
12/04/2013
(1 row affected)

As I though, it was working just fine. I suspected it had something to do with the custom date format, as EMC/OpenText tends to forget that not everyone uses ANSI or the american date format, so I decided to open a SR.

After some exchange of emails, we were told that both r_creation_date and r_modify_date where reserved words and that it was expected to fail. What???

After asking support to either fill this as a product limitation or fixing it, I decided to take a look at the code myself.

As I suspected, the query works just fine, and results are returned to the Query controller. The problem here is with the way REST generates the response. If you run the query with a different alias to see the result page you’ll get this:

"entries": [  {
"id": "http://127.0.0.1:8080/dctm-rest/repositories/repo.json?dql=select%20DATETOSTRING_LOCAL(r_creation_date,%27dd/mm/yyyy%27)%20as%20rr_creation_date%20from%20dm_document%20enable%20(return_top%201)&index=0",
"title": "12/04/2013",
"updated": "2017-06-16T11:04:38.284+00:00",
"published": "2017-06-16T11:04:38.284+00:00",
"content": {
"json-root": "query-result",
"definition": "http://127.0.0.1:8080/dctm-rest/repositories/repo/types/dm_document.json",
"properties": {
"rr_creation_date": "12/04/2013"
}}}]

Do you see those fields before the content element? That’s where everything breaks, why? Because of this:

public Date entryUpdated() {
Date updated = null;
Object modifyDate = ((QueryResultItem)getDataInternal())
  .getMandatoryAttribute("r_modify_date");
if (modifyDate == null) {
updated = new Date();
} else if ((modifyDate instanceof Date)) {
updated = (Date)modifyDate;
} else {
updated = DateFormatter.parse(modifyDate.toString());
}
return updated;
}

public Date entryPublished()
{
Date published = null;
QueryResultItem queryResultItem=getDataInternal();
Object modifyDate = ((QueryResultItem)getDataInternal())
  .getMandatoryAttribute("r_creation_date");
if (modifyDate == null) {
published = new Date();
} else if ((modifyDate instanceof Date)) {
published = (Date)modifyDate;
} else {
published = DateFormatter.parse(modifyDate.toString());
}
return published;
}

As you can see (besides the obvious copy/paste from one method to another changing the name of one variable), the standard view looks in the results from the query for a column with those names, and if it founds a column matching that name, tries to parse it with a forced format (which of course, it’s miserably failing with our custom date format).

This, in my opinion, is a bug, because the behaviour of the query functionality it is inconsistent between the Documentum stack, in fact, this query only fails with REST services, so I’ll keep pushing the SR to be treated as a bug and fixed by the talented team, and not as a “product limitation” or a “feature request”.

And if you face the same problem, and it is still not fixed, you have three options:

  • Keep waiting for a fix
  • Extend com.emc.documentum.rest.view.impl.QueryResultItemView and handle the IllegalArgumentException that throws DateFormatter.parse
  • Extend the controller adding a custom view for the results and removing/overriding the updated/published fields.

Filename in Documentum REST Services

If you have used Document REST Services you most likely have realized that downloading any content from an object return a “document”/”response” + dot + file extension.

You can check William Zhou’s answer here:

This had been discussed in the initial implementation but hadn’t been implemented since Content-Disposition response header is neither mandatory nor handled by all HTTP clients. Besides, it has some overhead to map the format/mime to a filename extension. But I think it has values helping for the download experience. It is appreciated if you can file a feature request CR so that we can discuss this with the product manager.

I’ve alredy raised a SR to support in order to get OpenText to consider adding this “feature”.

However, if you can’t/don’t want to wait, just “extend” com.emc.documentum.rest.controller.ContentMediaController adding the following line to the getContent method just before returning the response:

headers.setContentDispositionFormData("attachment", (String)co.getAttributeByName("object_name"));

and you’re good to go.