chimera tool quickly times out when chown -Ring large trees

[calestyo]

Hey.

When doing chimera chown -R on a large dir tree (hoping it would be faster than POSIX chown via NFS)... it times out after only a few seconds:

# chown -R 11001:11000 data12_calib
ERROR - Could not get connection to database
java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at org.dcache.db.AlarmEnabledDataSource.getConnection(AlarmEnabledDataSource.java:146)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at org.dcache.chimera.JdbcFs.inTransaction(JdbcFs.java:240)
	at org.dcache.chimera.JdbcFs.setInodeAttributes(JdbcFs.java:1050)
	at org.dcache.chimera.FsInode.setStat(FsInode.java:371)
	at org.dcache.chimera.cli.Shell$ChownCommand.chownRecursively(Shell.java:284)
	at org.dcache.chimera.cli.Shell$ChownCommand.chownRecursively(Shell.java:287)
	at org.dcache.chimera.cli.Shell$ChownCommand.chownRecursively(Shell.java:287)
	at org.dcache.chimera.cli.Shell$ChownCommand.call(Shell.java:266)
	at org.dcache.chimera.cli.Shell$ChownCommand.call(Shell.java:203)
	at org.dcache.util.cli.AnnotatedCommandExecutor.execute(AnnotatedCommandExecutor.java:126)
	at org.dcache.util.cli.CommandInterpreter$CommandEntry.execute(CommandInterpreter.java:268)
	at org.dcache.util.cli.CommandInterpreter.doExecute(CommandInterpreter.java:187)
	at org.dcache.util.cli.CommandInterpreter.command(CommandInterpreter.java:174)
	at org.dcache.util.cli.ShellApplication.execute(ShellApplication.java:149)
	at org.dcache.util.cli.ShellApplication.console(ShellApplication.java:197)
	at org.dcache.util.cli.ShellApplication.start(ShellApplication.java:86)
	at org.dcache.chimera.cli.Shell.main(Shell.java:96)
Bug detected! Please email the following details to <[email protected]>:
org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:309)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.startTransaction(AbstractPlatformTransactionManager.java:400)
	at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
	at org.dcache.chimera.JdbcFs.inTransaction(JdbcFs.java:240)
	at org.dcache.chimera.JdbcFs.setInodeAttributes(JdbcFs.java:1050)
	at org.dcache.chimera.FsInode.setStat(FsInode.java:371)
	at org.dcache.chimera.cli.Shell$ChownCommand.chownRecursively(Shell.java:284)
	at org.dcache.chimera.cli.Shell$ChownCommand.chownRecursively(Shell.java:287)
	at org.dcache.chimera.cli.Shell$ChownCommand.chownRecursively(Shell.java:287)
	at org.dcache.chimera.cli.Shell$ChownCommand.call(Shell.java:266)
	at org.dcache.chimera.cli.Shell$ChownCommand.call(Shell.java:203)
	at org.dcache.util.cli.AnnotatedCommandExecutor.execute(AnnotatedCommandExecutor.java:126)
	at org.dcache.util.cli.CommandInterpreter$CommandEntry.execute(CommandInterpreter.java:268)
	at org.dcache.util.cli.CommandInterpreter.doExecute(CommandInterpreter.java:187)
	at org.dcache.util.cli.CommandInterpreter.command(CommandInterpreter.java:174)
	at org.dcache.util.cli.ShellApplication.execute(ShellApplication.java:149)
	at org.dcache.util.cli.ShellApplication.console(ShellApplication.java:197)
	at org.dcache.util.cli.ShellApplication.start(ShellApplication.java:86)
	at org.dcache.chimera.cli.Shell.main(Shell.java:96)
Caused by: java.sql.SQLTransientConnectionException: HikariPool-1 - Connection is not available, request timed out after 30000ms.
	at com.zaxxer.hikari.pool.HikariPool.createTimeoutException(HikariPool.java:696)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:181)
	at com.zaxxer.hikari.pool.HikariPool.getConnection(HikariPool.java:146)
	at com.zaxxer.hikari.HikariDataSource.getConnection(HikariDataSource.java:100)
	at org.dcache.db.AlarmEnabledDataSource.getConnection(AlarmEnabledDataSource.java:146)
	at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:265)
	... 18 more

Cheers,
Chris.

[kofemann]

Looks like db connections don't play well with recursion.