Documentum in the cloud

Yesterday I attended the latest Documentum webex from Opentext. It was a review of  last year, with some use cases explained by customers and partners, and a review of the roadmap (It seems that they aren’t delaying the new Documentum version, and that it will be released by the end of this month).

Some of the topics/questions covered webtop (of course :D), new features such as iHub (a kind of Jasper Reports for Documentum, including an eclipse-based report designer), a D2 mobile application, and news regarding integration with the cloud.

It looks like the original idea EMC had of connectnig LEAP/cloud with Documentum through services is being pushed with a “common layer” of access in the form of services.

My idea of cloud may be completely wrong, but I don’t consider cloud a “remote datacenter”. Cloud is related to the concept of elasticity: elastic resources, usage and pricing. This means that you could dinamycally increase/decrease available resources according to your needs in a particular moment in time. In order to do this, you don’t just need to “go to the cloud”, you need your software to be able to do that (and Documentum is not able to).

I’m not going to discuss about microservices or containers, but a quick, clear example could be this one: Imagine you have a web application. You allocate one server on the cloud. In the cloud, you could, by defining threshold and behaviours, deal with peak workloads by automatically deploying additional server instances, that will automatically balance the load between them and then return to the “normal” usage without interaction. And this  would be extremely complex with Documentum (I guess you’ll need to configure a certain number of Documentum servers in HA, then stop them, and you could only deploy as many servers as you have previously configured in order to deal with high demand).

So, having discarded the viability of a “pure” cloud Documentum solution (note that I’m not including LEAP here, just the traditional Documentum server), we have two “offers” available:

IAAS: As far as I know, Opentext/Ondemand doesn’t have an IAAS offering available, so the alternative is to go to your favourite cloud provider, buy resources and then deploy Documentum there. However, you won’t get support (I guess this is something to be expected, as Opentext has its own cloud offering). Depending on your case, it might be worth “losing” support if figures make sense (same happened when VM were becoming popular, EMC didn’t support Documentum virtualized, but almost everyone was running it on virtual machines)

SAAS: Rather than cloud, this is a “remote datacenter” offering. Ondemand has fixed prices for this model, and the rest of competitors (if there are others left offering Documentum as a service) follow the same pricing model. This cloud offering, on paper, has the following advantages:

  • Lower TCO
  • Expert administration
  • Quicker startup (what?)
  • Better security (lol)
  • Disaster recovery with SLA response time

But you should have in mind also the following situations:

  • No specific ticketing application. Any operation that needs to be performed has to go through the support page (this includes, but it’s not limited to: creating a registered table, deploying a dar file, accesing dictionary information in d2, issuing SQL queries, running jobs, etc.).  In my experience, agile and rapid development cycles are not compatible with this way of working. Also note that the current Documentum SLAs always use “first response time” and not “time to solve the ticket” as metrics.
  • Better security: Well, if your product has security holes, the hole is there, no matter if it is on premise or on the cloud.
  • Disaster recovery: This is something I always say to clients: If you invest on a cloud environment, first thing to do is destroying the environment as soon as possible and request a recovery. This way you can have an accurate idea of how the provider works.

 

Opentext Documentum is coming next month

I didn’t realize that roadmap documents were updated last month. It looks like the February release is still going to happen (and I’ve been told a definite date, so it looks it won’t be delayed). After reviewing them (haven’t seen any changes :D), I can say:

  • Not much features regarding CS, the pattern/usage visualization and the s3 support (which, as far as I know, can be already done without official support) are the new features.
  • No word about DFS, and I know for a fact that several customers have actively asked for updates to current libs and extended support for application servers.
  • Clients get barely any changes (D2, Webtop, xCP).

I’m curious to see how many bugs are found in this first release from Opentext (and the brave customers that go first into the unknown :D), considering that some of the experienced Documentum staff left the company and the changes to the development cycle (that I guess happen when you move from a hardware company to a software company)

D2 configuration vs. a_is_hidden

Recently we came across a requirement that involved showing the a_is_hidden attribute in the content widget, regardless of its value.

At first glance, it seemed an easy goal to achieve, just add the attribute to the column list and it will work. Well, it doesn’t. The content widget just displays objects with a_is_hidden=false (duh!).

Next step, would be to find the configuration that manages this behaviour. Well, after looking a bit, we found the “configuration” that handles this:

com.emc.d2fs.dctm.content.FolderContent generates a DQL in order to retrieve the contents of the folder, and the method that generates this query uses com.emc.d2.api.preferences.ID2cPreferences to retrieve different preferences related to queries.

Some decompiling get us to the configurable side of things:

com.emc.d2.api.preferences.D2cPreferences:

 public DqlFilter getDqlFilter()
{
DqlFilter result = new DqlFilter(false, false);
return result;
}

com.emc.d2.api.preferences.DqlFilter:

public final class DqlFilter
{
private boolean m_showHidden;
private boolean m_showArchive;

public DqlFilter(boolean showHidden, boolean showArchive)
{
this.m_showHidden = showHidden;
this.m_showArchive = showArchive;
}

public String toString()
{
StringBuffer result = new StringBuffer();
if (!this.m_showHidden) {
result.append("a_is_hidden = false");
}
if (!this.m_showArchive)
{
if (result.length() != 0) {
result.append(" and ");
}
result.append("a_archive = false");
}
return result.toString();
}
}

So much for configurable application…

Documentum 16.3 delayed until Feb 2018

If you access the new support page, there is a section where you can find roadmaps for the products, and if you check the Documentum related ones, you’ll see the target date for ths new version listing new features. Without going into much detail (As I’m not sure how public this info is), these caught my atention:

  1. Ongoing exposure of D2 APIs through REST (goodbye DFS?)
  2. Native support for S3 object storage protocols to alleviate storage costs (goodbye OnDemand, hello AWS?)
  3. New REST Services (see #1, no mention of DFS anywhere to be found)
  4. Webtop will get a 16.3 version (yeah!)

 

Debugging D2 4.5/4.6

It still surprises me when people tell me that they can “only” debug D2 code by adding output to their plugins so, if you’re using eclipse as your IDE, here are some steps that most likely will help you to debug code from an eclipse server (Tomcat):

Extract all files from D2.war and D2-Config.war (just in case you want to have access to D2-Config from a local environment, not much you can debug here) and copy the D2 lib folder to your computer.

  • Generate lockbox file for your environment (probably this is the hardest part :D)
  • Update dfc.properties in both.
  • Update D2’s D2FS.properties (update absolute path in lockboxPath) and logback.xml (update logs path)
  • Update D2-Config’s D2-Config.properties (update absolute path in lockboxPath) and logback.xml (update logs path)
  • Copy following libraries to Tomcat folder:
    • /endorsed: (from d2-config/web-inf/lib)
      • crypto.jar
      • cryptojce.jar
      • cryptojcommon.jar
      • jcm.jar
      • jcmandroidfips.jar
      • jcmFIPS.jar
      • util.jar
    • /lib:
      • LB.jar
      • LBJNI.jar
  • Configure a new server in eclipse. In Run Configuration add the following:
    • Arguments:
      • -Djava.endorsed.dirs=”<absolute path to tomcat>\endorsed
      • -Djava.io.tmpdir=”<absolute path to some temp folder>\d2″
      • -Dclb.library.path=”<absolute path to D2 libraries folder>\lockbox\win_vc100_x64″ (choose your OS library)
    • Classpath:
      • User entries:
        • Path to LB.jar and LBJNI.jar
      • Environment:
        • Classpath: <absolute path to D2 libraries folder>\LB.jar;<absolute path to D2 libraries folder>\LBJNI.jar;<absolute path to D2 libraries folder>\C6-Common.jar;<absolute path to Documentum DFC’s dctm.jar>;<absolute path to Documentum config folder>
        • Path: <absolute path to D2 libraries folder>\lockbox\win_vc100_x64;%PATH%
    • Add project to Sources
  • Add D2 and D2-Config as deployed modules to your server configuration.
  • Start server

Support adventures II (REST Services edition)

I wasn’t expecting this week to be so “productive” 🙂

I had the chance to engage again with Documentum support, which is always an adventure 😉

We’re currently porting a framework from DFS to REST services. A couple of days ago, one of my colleagues had a problem and ask me if I knew why a query through the REST services was failing.

The query is using DATETOSTRING to retrieve a time field and returning a column with the attribute name using the alias, so, if a user requests expiration_date we execute a query like:

select DATETOSTRING(expiration_date,’dd/mm/yyyy’) as expiration_date from…

which returns the requested attribute in the desired (configurable) format. Easy, right? Well, the problem was that using r_creation_date or r_modify_date was throwing an exception:

 {    “status”: 400,
“code”: “E_INPUT_ILLEGAL_ARGUMENTS”,
“message”: “There are illegal arguments provided.”,
“details”: “Invalid format: \”12/04/2013\” is malformed at \”/04/2013\””}

We also observed that changing the alias to something else than r_creation_date/r_modify_date would work (but it was breaking our use case).

This was weird, because I was quite sure that those kind of queries worked previously, so I checked against the Content Server:

Connected to Documentum Server running Release 7.1.0210.0328  Linux64.Oracle
1> select DATETOSTRING(r_creation_date,’dd/mm/yyyy’) as r_creation_date from dm_document enable (return_top 1);
2> go
r_creation_date
—————
12/04/2013
(1 row affected)

As I though, it was working just fine. I suspected it had something to do with the custom date format, as EMC/OpenText tends to forget that not everyone uses ANSI or the american date format, so I decided to open a SR.

After some exchange of emails, we were told that both r_creation_date and r_modify_date where reserved words and that it was expected to fail. What???

After asking support to either fill this as a product limitation or fixing it, I decided to take a look at the code myself.

As I suspected, the query works just fine, and results are returned to the Query controller. The problem here is with the way REST generates the response. If you run the query with a different alias to see the result page you’ll get this:

"entries": [  {
"id": "http://127.0.0.1:8080/dctm-rest/repositories/repo.json?dql=select%20DATETOSTRING_LOCAL(r_creation_date,%27dd/mm/yyyy%27)%20as%20rr_creation_date%20from%20dm_document%20enable%20(return_top%201)&index=0",
"title": "12/04/2013",
"updated": "2017-06-16T11:04:38.284+00:00",
"published": "2017-06-16T11:04:38.284+00:00",
"content": {
"json-root": "query-result",
"definition": "http://127.0.0.1:8080/dctm-rest/repositories/repo/types/dm_document.json",
"properties": {
"rr_creation_date": "12/04/2013"
}}}]

Do you see those fields before the content element? That’s where everything breaks, why? Because of this:

public Date entryUpdated() {
Date updated = null;
Object modifyDate = ((QueryResultItem)getDataInternal())
  .getMandatoryAttribute("r_modify_date");
if (modifyDate == null) {
updated = new Date();
} else if ((modifyDate instanceof Date)) {
updated = (Date)modifyDate;
} else {
updated = DateFormatter.parse(modifyDate.toString());
}
return updated;
}

public Date entryPublished()
{
Date published = null;
QueryResultItem queryResultItem=getDataInternal();
Object modifyDate = ((QueryResultItem)getDataInternal())
  .getMandatoryAttribute("r_creation_date");
if (modifyDate == null) {
published = new Date();
} else if ((modifyDate instanceof Date)) {
published = (Date)modifyDate;
} else {
published = DateFormatter.parse(modifyDate.toString());
}
return published;
}

As you can see (besides the obvious copy/paste from one method to another changing the name of one variable), the standard view looks in the results from the query for a column with those names, and if it founds a column matching that name, tries to parse it with a forced format (which of course, it’s miserably failing with our custom date format).

This, in my opinion, is a bug, because the behaviour of the query functionality it is inconsistent between the Documentum stack, in fact, this query only fails with REST services, so I’ll keep pushing the SR to be treated as a bug and fixed by the talented team, and not as a “product limitation” or a “feature request”.

And if you face the same problem, and it is still not fixed, you have three options:

  • Keep waiting for a fix
  • Extend com.emc.documentum.rest.view.impl.QueryResultItemView and handle the IllegalArgumentException that throws DateFormatter.parse
  • Extend the controller adding a custom view for the results and removing/overriding the updated/published fields.

Updated roadmap, webinar and FAQ from OpenText Documentum

If you registered for the webinar OpenText held a few weeks ago (if you didn’t, maybe you can check Andrey’s post on the subject), you should have received an invitation to some site from OT with a FAQ about the Documentum stack. I’m not going to paste it here, just in case it’s not public, but IMHO, the highlights are:

  • Content Server: Aligment with ECD. No new features but trying to move to a microservices architechture. It probably means CS won’t evolve anymore.
  • Webtop: Several points on this. Will keep being supported and updated to support latest CS.
  • Rest: Every new product based on this. No mention about DFS (I hope it’s dead, unless they decide to update with libraries from this decade)
  • xCP: It looks like xCP 1.x support will be over by the end of 2019.
  • Support: Still through EMC’s site until the end of the year.