Multiple environments with Tomcat/DA

Similar to the previous post about configuring dqMan/DQLTester (Multiple environments with dqMan/DQLTester), a Tomcat with DA can be configured the same way:

Folder structure:

  • da.bat
  • bin
  • conf
  • jre
  • lib
  • logs
  • webapps
  • properties
    • env1
      • dev
        • dfc.properties
        • dmcl.ini
      • prod
        • dfc.properties
        • dmcl.ini
    • env2
      • dev
        • dfc.properties
        • dmcl.ini
      • prod
        • dfc.properties
        • dmcl.ini
  • etc.

DA.bat:

@Echo off
SETLOCAL ENABLEEXTENSIONS
SET folderbase=dfcproperties

echo ****************
echo 1. env1 dev
echo 2. env2 prod
...
echo ****************

SET /p var= ^> Choose option: 

if "%var%"=="1" goto op1
if "%var%"=="2" goto op2
...

:op1
SET foldervar=%folderbase%\env1\dev
goto finish

:op2
SET foldervar=%folderbase%\env1\prod
goto finish
...

:finish
set JAVA_HOME=%cd%\jre
set JAVA_OPTS=-Ddfc.properties.file=%cd%\%foldervar%\dfc.properties -Dfile.encoding=UTF-8 -Xms128m -Xmx512m -XX:+UseG1GC -Djava.net.preferIPv4Stack=true -Djava.net.preferIPv4Addresses=true -Djava.net.preferIPv6Addresses=false -Djava.net.preferIPv6Stack=false

start "" /D %cd%\bin /B %cd%\bin\catalina.bat start
cls&exit

Mozilla To Drop Support For All NPAPI Plugins In Firefox 52 Except Flash

https://tech.slashdot.org/story/17/02/04/2323204/mozilla-to-drop-support-for-all-npapi-plugins-in-firefox-52-except-flash

What does this mean to Documentum? Well, if your Webtop/D2 is using applets (UCF), starting March, you’ll only be able to use it with IE11 and older versions, as no other major browser (actually, Firefox is the only “major” browser that still allows NPAPI plugins, but this is due to their slow pace of development) will allow NPAPI plugins (Java).

D2 (and Webtop and xCP) CTF

CTF (Content Transfer Framework) is how EMC Dell calls their “new UCF”. It works as a browser extension, and is the same extension you’ve used for the latest version of Webtop (new functionality getting first to Webtop? LOL). And this mode is not the default (why?) so you’ll need to change it in the settings.properties file of D2.

Also, this extension will generate some “index” files on the folder where you download files:

  • .checkout.xml
  • .d2_edit_storage.json
  • .d2_view_storage.json
  • .view.xml

That contain object names, ids, operation performed, and folder paths of the files transferred.

Tested on latest Firefox Nightly x64 and Chrome.

FYI, I’m pasting the “wonderful” ASCII compatibility matrix provided by Dell in the configuration file:

#     +——————-+——+——+——+——+
#     | Browser:OS \ Mode | Thin | Java | ctf  | Note |
#     +——————-+——+——+——+——+
#     | IE 11             | yes  | yes  | yes  |      |
#     +——————-+——+——+——+——+
#     | Edge              | yes  | NO   | NO   | (1)  |
#     +——————-+——+——+——+——+
#     | Firefox           | yes  | yes  | yes  |      |
#     +——————-+——+——+——+——+
#     | Chrome            | yes  | NO   | yes  | (1)  |
#     +——————-+——+——+——+——+
#     | Safari:Mac_OSX    | yes  | yes  | yes  |      |
#     +——————-+——+——+——+——+
#     | Safari:Mac_IOS    | yes  | NO   | NO   | (2)  |
#     +——————-+——+——+——+——+
# Notes:
#
# (1) Chrome and Edge do not support java applets, and Edge does not support the CTF plugin.
#     D2 will fallback to thin client mode appropriately when java or ctf has been
#     specified in the value of the browser.plugin.mode setting as described above.
#
# (2) If browser.plugin.mode contains java or ctf, then D2 will silently continue to run in
#     thin client mode. Safari running on Mac_IOS does not support the java or ctf plugin.

  • Prompt for installing the extension:

ctf-prompt

  • View content prompt:

ctf-save

  • Edit:

ctf-downloading

  • Check in:

ctf-uploading

What I expect for Documentum from ECD/OpenText

What I think OpenText should keep:

  • Support Community
  • ContentHub
  • EMC Support (at least the search engine with the knowledge base)
  • Download Center (After so many changes, I really don’t want to change again :P)
  • EMC GitHub
  • Webtop (It’s the only client that works OOTB)

What I think OpenText should change:

  • Sales policy (Stop pushing products that customers don’t need, and yes, that means D2 and xCP for customers that simply want a “library”)
  • Sort libraries/dependencies in the Documentum stack (if 7.x is released, make it that every product has the same version of every library). Rebrand products accordingly (yeah, Webtop 7.x if it is using DFC 7.x)
  • Detailed changelog of the products (don’t force us to go through every single customization to check if some component we’re extending has changed and broke the customizations)
  • Kill lockbox with fire (or at least, make it optional for both CS and D2). Nobody understands it, nobody knows how to set it up, so it is more a pain in the ass than a security feature
  • Oh, and remove D2 dependency on JMS

I’ve lost any hope to see dmbasic gone for good, or JBOSS replaced with something “lighter”, that’s why those didn’t make it into the list (as probably many others, that I don’t remember now)

(And yes, I’m not a huge fan of D2 :P)

WebTop’s new content transfer mechanism

Yesterday I saw Webtop 6.8.2 was released, and as this was great news because this version was supposed to remove Java applets once and for all. While most of us though that removing applets would imply moving to HTML5, EMC has decided to implement an intermediate solution 😦

Webtop 6.8.2 will prompt users to install a browser extension first time you log in:

webtop-warning

In Firefox the extension will be installed, and in Chrome user will be redirected to the Chrome Web Store:

webtop-chrome-extension

After installing the extension, the first time users try to download something they’ll be prompted to install a local application that will handle the transfers (WebSockets I presume).

Good news: Now you don’t need to fight with Java/browser combination

Bad news: Users have to perform operations. Not the cleanest solution (IMHO).

Bonus: Now importing files is really not-user friendly:

  1. File -> import
  2. Ugly Java window will pop-up over the import container asking for files to import
  3. Select files/folders, press ok and… nothing happens. You are kept in a blank container window with “Next” and “Finish” buttons and no indication whatsoever if the files are been/have been uploaded or what is going on.

ESAPI vs character codification

Webtop Patch notes state that problems with esapi have been fixed:

WEBTOP-32460
Opening a document with accent in name using http mode with Webtop 6.8 using IE11 results in a security exception and the contents of the file are not displayed.

Well, if you check HttpTransportManager.class you can check the “fix” by yourself:

Prepatch:

    try{
       contentDisposition.append(SECURITY.validator().getValidHeader("HttpTransportManager", makeSafeHeaderValue(strCleanFileName), "Header Manipulation"));
    }catch (UnsupportedEncodingException e){
        throw new WrapperRuntimeException(e);
    }

Patch “fix”:

    try{
         ClientInfo localClientInfo = ClientInfoService.getInfo();
         if ((localClientInfo.isPlatform(ClientInfo.WIN)) && (localClientInfo.isBrowser(ClientInfo.MSIE))) {
            contentDisposition.append(makeSafeHeaderValue(str));
          } else {
            contentDisposition.append(SECURITY.validator().getValidHeader("HttpTransportManager", makeSafeHeaderValue(str), "Header Manipulation"));
          }
    }catch (UnsupportedEncodingException localUnsupportedEncodingException){
          throw new WrapperRuntimeException(localUnsupportedEncodingException);
    }

This is obviously a lazy way to “fix” it, as the problem is in the ESAPI library and skipping the security check is not really a way to fix it.

There are basically 3 problems with the character enconding in wdk:

  1. The way WDK handles characters: This is a minor problem, as even if WDK does weird things such as converting back and forth the charecters, it “works”
  2. ESAPI is not character encoding aware, this makes useless adding your “special” characters to the ESAPI.properties validators, as this would be read incorrectly by ESAPI (more info here: Virtuallinks vs character encoding vs ESAPI)
  3. Inconsistencies when WDK calls ESAPI security validators as it sends the string to validate in different encodings (escaped/non-escaped, UTF-8 “native”, UTF-8 “converted”, etc.).

So, in order to fix this problems we need to fix the ESAPI library to be aware of the character encoding:

1. Get ESAPI 2.1.0 sources (Downloads – owasp-esapi-java – OWASP Enterprise Security API (Java Edition) – Google Project Hosting)

2. Add the following entries to webtop/WEB-INF/ESAPI.properties:

Validator.InputEncoding=UTF-8 //App server encoding
Validator.OutputEncoding=ISO-8859-1 //Locale encoding

3. Modify Validator.HTTPHeaderValue, Validator.FileName and Validator.DirectoryName with your special characters:

Validator.HTTPHeaderValue=^[a-zA-Z0-9<strong>áéíóúÁÉÍÓÚñÑ</strong>()\\-=\\*\\.\\?;,+\\/:&amp;_ ]*$
Validator.FileName=^[a-zA-Z0-9<strong>áéíóúÁÉÍÓÚñÑ</strong>!@#$%^&amp;{}\\[\\]()_+\\-=,.~'` ]{1,255}$
Validator.DirectoryName=^[a-zA-Z0-9<strong>áéíóúÁÉÍÓÚñÑ</strong>:/\\\\!@#$%^&amp;{}\\[\\]()_+\\-=,.~'` ]{1,255}$

4. Modify Validator.WDKHTTPURI adding a blank (because EMC assumes that nobody uses blanks when naming a folder):

Validator.WDKHTTPURI=^/([a-zA-Z0-9. \\-_]*/?)$

5. Add the following lines to org.owasp.esapi.reference.DefaultSecurityConfiguration

    private String inputEncoding = null;
    private String outputEncoding = null;  

    public String getInputEncoding() {
        return getESAPIProperty(INPUT_ENCODING, "UTF-8"); //UTF-8 is the default value returned if Validator.InputEncoding is not found
    }  

    public String getOutputEncoding() {
        return getESAPIProperty(OUTPUT_ENCODING, "ISO-8859-1"); //ISO-8859-1 is the default value returned if Validator.OutputEncoding is not found
    }  

    private String getEncodedESAPIProperty(String key){
        try {
            if (inputEncoding!=null && outputEncoding!=null){
                return new String(((String)properties.get(key)).getBytes(outputEncoding), inputEncoding);
            }else{
                return (String)properties.get(key);
            }
        } catch (UnsupportedEncodingException e) {
            e.printStackTrace();
        }
        return key;
    }

6. Modify DefaultSecurityConfiguration(), getESAPIProperty(String key, String def) and getESAPIProperty(String key, boolean def) methods:

    public DefaultSecurityConfiguration() {
        // load security configuration
        try {
            loadConfiguration();
            this.setCipherXProperties();  

            //deal with encoding
            inputEncoding=getInputEncoding();
            outputEncoding=getOutputEncoding();  

        } catch( IOException e ) {
          logSpecial("Failed to load security configuration", e );
          throw new ConfigurationException("Failed to load security configuration", e);
        }
    }  

    protected String getESAPIProperty( String key, String def ) {
        //String value = properties.getProperty(key);
        String value=getEncodedESAPIProperty(key);
        if ( value == null ) {
              logSpecial( "SecurityConfiguration for " + key + " not found in ESAPI.properties. Using default: " + def, null );
              return def;
        }
        return value;
    }  

    protected boolean getESAPIProperty( String key, boolean def ) {
        //String property = properties.getProperty(key);
        String property=getEncodedESAPIProperty(key);
        if ( property == null ) {
              logSpecial( "SecurityConfiguration for " + key + " not found in ESAPI.properties. Using default: " + def, null );
              return def;
        }
        if ( property.equalsIgnoreCase("true") || property.equalsIgnoreCase("yes" ) ) {
            return true;
        }
        if ( property.equalsIgnoreCase("false") || property.equalsIgnoreCase( "no" ) ) {
            return false;
        }
        logSpecial( "SecurityConfiguration for " + key + " not either \"true\" or \"false\" in ESAPI.properties. Using default: " + def, null );
        return def;
    }

7. Now that you are modifying this class, you can comment the logSpecial calls from loadPropertiesFromStream,loadConfiguration and getResourceFile and skip the messages thrown by ESAPI when loading the libraries (Best Practices – Review before releasing)

8. Modify org.owasp.esapi.reference.validation.StringValidationRule:

private String checkWhitelist(String context, String input, String orig) throws ValidationException{
  // check whitelist patterns
  //deal with encoding
  DefaultSecurityConfiguration sec=(DefaultSecurityConfiguration)DefaultSecurityConfiguration.getInstance();  

  Charset inputcharset = Charset.forName(sec.getInputEncoding());
  Charset outputcharset = Charset.forName(sec.getOutputEncoding());  

  ByteBuffer inputBuffer=null;
  try {
      inputBuffer = ByteBuffer.wrap(URLDecoder.decode(input,sec.getOutputEncoding()).getBytes());
  } catch (UnsupportedEncodingException e) {
      e.printStackTrace();
  }  

  CharBuffer data = inputcharset.decode(inputBuffer);  

  ByteBuffer outputBuffer = outputcharset.encode(data);
  byte[] outputData = outputBuffer.array();
  input=new String(outputData);  

    for (Pattern p : whitelistPatterns) {
        if ( !p.matcher(input).matches() ) {
            throw new ValidationException( context + ": Invalid input. Please conform to regex " + p.pattern() + ( maxLength == Integer.MAX_VALUE ? "" : " with a maximum length of " + maxLength ), "Invalid input: context=" + context + ", type(" + getTypeName() + ")=" + p.pattern() + ", input=" + input + (NullSafe.equals(orig,input) ? "" : ", orig=" + orig), context );
        }
    }  

    return input;
}

9. Generate jar file or replace the classes in the bundled esapi.jar.

This changes will work for virtual links and http transfer, however you may need to modify additional methods/validators depending on your customizations or case uses, but you get the idea.

This “patch” works with webtop 6.8 latest patch in every browser I’ve tested (ie, firefox, chrome, opera).