Hello Dimitar,
There is no direct way of specifying a timeout for a destination or a function module invocation. But still, it is possible to limit the execution time of function modules by limitting the life span of the session. As soon as the session is ending and JCo is recognizing this in its runtime, it will cancel all function module executions belonging to this session and is thus freeing the resources on both the ABAP system an the JCo side. For that you need a fitting implementation of JCoSessionReferenceProvider that fits to your purposes.
BTW, the hanging program is simply waiting for the response of the ABAP function module - you could als verify whether the function module code could be imporved. Last but not least, if you need to execute more function modules at the same time, you should consider to allow more in the destination configuration and increase the peak size.
Best regards,
Markus