Timeout exception while starting Liferay on Jboss EAP 7.x or Wildfly

Issue:

Jboss server throws following error while start up :

2018-07-23 21:36:09,29 [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("interface" => "management")]'

2018-07-23 21:36:09,35 [org.jboss.as.controller.management-operation] (Controller Boot Thread) JBAS013412: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[ ("core-service" => "management"), ("management-interface" => "http-interface") ]' 

2018-07-23 21:36:09,295 ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0190: Step handler org.jboss.as.server.DeployerChainAddHandler$FinalRuntimeStepHandler@47f29910 for operation {"operation" => "add-deployer-chains","address" => []} at address [] failed handling operation rollback -- java.util.concurrent.TimeoutException: java.util.concurrent.TimeoutException
        at org.jboss.as.controller.OperationContextImpl.waitForRemovals(OperationContextImpl.java:511)
        at org.jboss.as.controller.AbstractOperationContext$Step.handleResult(AbstractOperationContext.java:1369)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeInternal(AbstractOperationContext.java:1328)
        at org.jboss.as.controller.AbstractOperationContext$Step.finalizeStep(AbstractOperationContext.java:1301)
        at org.jboss.as.controller.AbstractOperationContext$Step.access$300(AbstractOperationContext.java:1185)
        at

Liferay is a big application running on jboss server and it may take 5-10 minutes to start the servers. So how to set the timeout to 600 or 900 seconds ? Where and how to set it ?

For detail information please read the official document from redhat https://access.redhat.com/solutions/1190323

You can configure jboss.as.management.blocking.timeout system property to tune timeout (seconds) waiting for service container stability.

For standalone mode:

To resolve the problem by increasing the block management timeout to 600-900 in standalone.xml. The default timeout was set to 300 so liferay startup takes more than five minutes.

The above will set jboss.as.management.blocking.timeout system property in configuration file (i.e standalone-*.xml) like :

...
</extensions>
<system-properties>
      <property name="jboss.as.management.blocking.timeout" value="600"/>
</system-properties>
<management>
...

or

Add -Djboss.as.management.blocking.timeout=600 to jvm argument of standalone.sh or add the following JAVA_OPTS in standalone.conf:

JAVA_OPTS="$JAVA_OPTS -Djboss.as.management.blocking.timeout=600"

For Domain mode:

Edit the $JBOSS_HOME/bin/domain.conf and set it via this line -Djboss.as.management.blocking.timeout=600 after the if block:

JAVA_OPTS="$JAVA_OPTS -Djboss.as.management.blocking.timeout=600"

How to verify Private Key with a Certificate (SSL)

Private key contains series of numbers. Two numbers are for the Public key, others are for Private key.
Public key bits are also embedded in the Certificate.
To check that Public key in your Certificate matches the "Public key" inside your Private key, you need to view the Certificate & Private key and compare the numbers.

To view the Certificate and Private key please follow the commands.

# openssl x509 -noout -text -in domain.crt
# openssl rsa -noout -text -in domain.key

The 'modulus' and the 'public exponent' portions in the key and the Certificate must match. Human eye it is dificult to compare the 'public exponent', it is usually 65537. To compare these in shorter number follow the commands, it will show the hash.

# openssl x509 -noout -modulus -in domain.crt | openssl md5

Result: (stdin)= 065ae4a31a6f5623e86f5bc17532c7e3

# openssl rsa -noout -modulus -in domain.key | openssl md5

Result: (stdin)= cce154b2e767186b3e146af70101f046

Compare it in single command

# openssl x509 -noout -modulus -in domain.crt | openssl md5 ;\
  openssl rsa -noout -modulus -in domain.key | openssl md5

Result:
(stdin)= 065ae4a31a6f5623e86f5bc17532c7e3
(stdin)= cce154b2e767186b3e146af70101f046

You can also compare CSR with the Private key with following command

# openssl req -noout -modulus -in domain.csr | openssl md5

Result: (stdin)= cce154b2e767186b3e146af70101f046

Command to find your public IP.

If you are behind a router and don't know your public IP.
You could request some website to get your IP using curl or wget. 
Following commands will give you the result.
It returns just the plain text IP, nothing else

• curl http://checkip.amazonaws.com
• curl ipinfo.io/ip
• curl icanhazip.com or wget -qO- icanhazip.com
• curl ident.me; echo or curl v4.ident.me; echo
• wget -qO- http://ipecho.net/plain ; echo
• curl ipecho.net/plain ; echo
• curl -s checkip.dyndns.org | sed -e 's/.*Current IP Address: //' -e 's/<.*$//'
• curl ifconfig.me

Redirecting Hash(#) Fragments with Apache rewrite.

The character string beginning with a pound sign, # being interpreted as its HTML entity hex equivalent, %23..
It can led the url with 404 error.
It is very simple to redirect hash-fragment requests to proper destination using rewrite techniues.

You can find in the web access logs with HTML character code %23 for #.

Example1:

http://www.test.com/path/%23blog-123

To fix this you can do like this.

RewriteRule ^/(.*)%23blog-(.*)$ /$1#blog-$2 [R=301,L,NE]

In the given example  the first (.*) is used to replace the $1, and the second (.*) is used to replace the $2. The key here is the NE flag, which prevents # from being converted to its hex code of %23 during the rewrite.

Example2:

It is also easy to do general url redirection.
Let's assume you wanted to redirect url /web/tips-for-parents to /take-away-for-enablers/#familiestrack

RewriteRule  ^/web/tips-for-parents  /take-away-for-enablers/#familiestrack [R=301,L,NE]

Exaple3:

Lets redirect a part of the url which is appended as hash-fragment.

RewriteRule ^/path-of-url/(.*)/?$  /path-of-url/#$1 [R=301,L,NE]

So if request is for /path-of-url/newsite, it will redirected to /path-of-url/#newsite


NE|noescape : By default, special characters, such as & and ?, for example, will be converted to their hexcode equivalent. Using the [NE] flag prevents that from happening.

Apache flag NE

Overwrite Cache-Control headers in Liferay Document Library files

Manage Cache of Liferay document library files.
By default Liferay disable cache for all documents and images in its document library. To enable cache control follow the steps.

DLFileEntry from Document library are downloaded from WebServerServlet, that is adding Cache-Control: private header but no other caching instruction (like max-age or Expires) is appended.

ServletResponseUtil will override the Cache-Control header, no matter what was set before. In this particular case, when trying to modify the header using the HeaderFilter will fail because the portal will override it with a hard-coded value later on.

Overwrite Cache-Control headers settings for tomcat.

1. Shutdown the Liferay portal serer (Tomcat).
2. Edit the <tomcat_home>/webapps/ROOT/WEB-INF/liferay-web.xml
3. Add the following filter mapping:    
   

       <filter-mapping>
           <filter-name>Header Filter</filter-name>
           <url-pattern>/documents/*</url-pattern>
       </filter-mapping>
      
   I've added it after the following block:
      
       <filter-mapping>
           <filter-name>Header Filter</filter-name>
           <url-pattern>*.png</url-pattern>
       </filter-mapping>

     
                <filter-mapping>
                   <filter-name>Header Filter</filter-name>
                   <url-pattern>*.jpg</url-pattern>
                </filter-mapping>
   
   
4. Start the portal again.

Check Header detail in the browser.

Before configuration change 

Open the browser with cache cleaned. Also open network tool using (F12).

Try open the URL which has document library file. 

URL: https://localhost/documents/20623/76583/resw-fty.jpg/6134e354-56b9-43b1-7650-9cc5230e7654?t=1234200776548
Result:  This should have "Cache-Control: private"

After configuration change

Open the browser with cache cleaned. Also open network tool using (F12).

Try open the URL which has document library file. 

URL: https://localhost/documents/20623/76583/resw-fty.jpg/6134e354-56b9-43b1-7650-9cc5230e7654?t=1234200776548
Result: This should have "Cache-Control:max-age=315360000, public"

 

Common Subversion Errors with Apache

Error: Can't open db/txn-current-lock - permission denied

This is one of the common error while accessing SVN via Apache httpd server.
This error comes due to misconfiguration of Apache and SVN
Here are the additional steps which you need to perform to resolve this issue.

1. Configure run apache with non root previledged user. 
2. Give read and write permission to that user on SVN repository folder

Example:

• useradd -s /sbin/nologin apache-user

• Change in httpd.conf
  User apache-userGroup apache-user

• Restart apache

Change user ownership and file permission on SVN repository folders

• chown -R apache-user /path/to/repository
• chmod -R 770 /path/to/repository

Error: Couldn't open rep-cache database" (post commit FS processing)

If you are using PHP on Apache which is loading SQLite can cause this issue.
This issue comes when any older version of SQLite modules loaded by Apache.
Check your Apache configuration and load mod_dav_svn before mod_php

Linux file-folder permission sticky bit

In UNIX like systems there are special permissions for folders and files. Sticky bit is one of the special permission.

The sticky bit is a user ownership access right flag that can be assigned to files and directories on Unix-like systems.

When a directory's sticky bit is set, the file system treats the files in such directories in a special way so only the file's owner, the directory's owner, or root user can rename or delete the file.

The sticky bit can be set using the chmod command and can be set using its octal mode 1000 or by its symbol t.

For example, to add the sticky bit on the directory /test one can type chmod +t /test
To clear it, use chmod -t /test

If the sticky-bit is set on a file or directory which doesn't have execution permission to others, it is indicated with a capital T