fabric continuously Transferring file: libfabric.so, version 3.8.1.3
Posted: Thu, 07/31/2008 - 13:37
fabric continuously Transferring file: libfabric.so, version 3.8.1.3
I cannot seem to successfully deploy an app to the fabric and utilize the app with a client app. I am attempting to deploy the spring bean hello world app, and it appears to successfully deploy to the fabric, but when I go to run the client app, it doesn't seem to do anything.
The error is as follows:
Buildfile: build.xml
setup:
run:
[java] Jul 31, 2008 9:57:50 AM org.springframework.context.support.AbstractApplicationContext prepareRefresh
[java] INFO: Refreshing org.springframework.context.support.ClassPathXmlApplicationContext@ecd7e: display name [org.springframework.context.support.ClassPathXmlApplicationContext@ecd7e]; startup date [Thu Jul 31 09:57:50 EDT 2008]; root of context hierarchy
[java] Jul 31, 2008 9:57:50 AM org.springframework.beans.factory.xml.XmlBeanDefinitionReader loadBeanDefinitions
[java] INFO: Loading XML bean definitions from class path resource [spring_hello_world_bean.xml]
[java] Jul 31, 2008 9:57:50 AM org.springframework.context.support.AbstractApplicationContext obtainFreshBeanFactory
[java] INFO: Bean factory for application context [org.springframework.context.support.ClassPathXmlApplicationContext@ecd7e]: org.springframework.beans.factory.support.DefaultListableBeanFactory@172e449
[java] Jul 31, 2008 9:57:50 AM org.springframework.beans.factory.support.DefaultListableBeanFactory preInstantiateSingletons
[java] INFO: Pre-instantiating singletons in org.springframework.beans.factory.support.DefaultListableBeanFactory@172e449: defining beans [localHelloWorldBean,fabricHelloWorldBean,fabric]; root of factory hierarchy
[java] Hello World, locally.
[java] Exception in thread "main" org.springframework.remoting.RemoteAccessException: Failed to invoke fabric stub for remote component; nested exception is com.appistry.fabric.TransactionException: The request was not attempted. 'Component and or method 'hello_world_component.greet(java.lang.String)' does not exist
[java] Caused by: com.appistry.fabric.TransactionException: The request was not attempted. 'Component and or method 'hello_world_component.greet(java.lang.String)' does not exist
[java] at com.appistry.spring.Interceptor.handleFabricTransactionException(Unknown Source)
[java] at com.appistry.spring.Interceptor.handleException(Unknown Source)
[java] at com.appistry.spring.Interceptor.executeRequest(Unknown Source)
[java] at com.appistry.spring.Interceptor.invoke(Unknown Source)
[java] at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:166)
[java] at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
[java] at $Proxy0.greet(Unknown Source)
[java] at com.appistry.samples.spring_hello_world.client.HelloWorldSpringClient.main(Unknown Source)
BUILD FAILED
/home/username/tutorial_samples/java/spring_hello_world/spring_hello_world_client/build.xml:33: Java returned: 1
Total time: 7 seconds
I am wondering if the constant transferring of the libfabric.so file has anything to do with this.
Bookmark this post with:
[java] Exception in thread "main" org.springframework.remoting.RemoteAccessException: Failed to invoke fabric stub for remote component; nested exception is com.appistry.fabric.TransactionException: The request was not attempted. 'Component and or method 'hello_world_component.greet(java.lang.String)' does not exist
[java] Caused by: com.appistry.fabric.TransactionException: The request was not attempted. 'Component and or method 'hello_world_component.greet(java.lang.String)' does not exist
abovill,
The section above makes me think that the hellow world fabric application did not load properly. One troubleshooting technique to determine why it didn't load is to look at the log_monitor traffic while a fabric worker is loading. Please follow these instructions to get log_monitor output and post the log here on the forums:
Second, the libfabric.so message repeating is odd. What exact operating system are you running on?
You know how you deleted that file before? You may have that same issue on another worker as well. Is the message telling you which worker is having the problem? If so, you may have to delete not only the file in the staging directory, but maybe also the one in the system directory on that box.
The log_monitor output may help us determine if these issues are related.
In addition, whenever I try to run a client, I get the following error:
Jul 31 09:30:11 00:11:43:CD:FA:A8 FabricProcess[E:3022] Could not get task service for 'e8c76c5f-a5ee-4ea1-81e7-bca94a89827c', state ''
Here is that log... note that before the log is started, all workers are stopped (and confirmed stopped)
----
(this stuff just repeated endlessly until I started the first worker)
Aug 01 04:08:50 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:16 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:02 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:07:27 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:18 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:08:52 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:04 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:07:29 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:05:06 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.47:40919
Aug 01 06:00:54 00:1E:C9:B2:4F:D1 FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 04:05:06 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.65:32777
Aug 01 04:05:06 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.63:32782
Aug 01 03:06:12 00:11:43:CE:0D:9A FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 04:09:00 00:11:43:CE:0C:D4 FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 04:05:06 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.62:32931
Aug 01 04:07:37 00:11:43:CE:0E:0C FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 04:05:06 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.64:32776
Aug 01 03:06:26 00:11:43:CE:0D:CE FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 06:00:54 00:1E:C9:B2:4F:D1 FabricKeeper[I:7124] Transferring file: libfabric.so, version 3.8.1.3
Aug 01 04:05:07 00:11:43:CD:FA:A8 FabricKeeper[I:7110] got new file (upload/libfabric.so) - running postprocessing
Aug 01 04:05:09 00:11:43:CD:FA:A8 FabricKeeper[I:7108] Spreading file: published libfabric.so
Aug 01 04:05:09 00:11:43:CD:FA:A8 FabricKeeper[I:7108] Spreading file: libfabric.so (3.8.1.3)
Aug 01 04:09:07 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:33 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:19 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:07:44 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:09:15 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:41 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:26 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:07:52 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:05:21 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.47:40939
Aug 01 06:01:09 00:1E:C9:B2:4F:D1 FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 06:01:09 00:1E:C9:B2:4F:D1 FabricKeeper[I:7124] Transferring file: libfabric.so, version 3.8.1.3
Aug 01 04:05:22 00:11:43:CD:FA:A8 FabricKeeper[I:7110] got new file (upload/libfabric.so) - running postprocessing
***AT THIS POINT WORKER POS1 WAS STARTED!***
Aug 01 04:05:26 00:11:43:CD:FA:A8 FAM[I:14000] FAM Member Service started on port(s) 47212
Aug 01 04:05:26 00:11:43:CD:FA:A8 FabricWorkerManager[I:10000] Location: 10.10.10.61 FabricWorkerManager: 239.255.0.75:32000 FabricWorkerManager initialized...
Aug 01 04:05:26 00:11:43:CD:FA:A8 FabricWorkerManager[I:10001] FabricWorkerManager OS path: /sbin:/usr/sbin:/bin:/usr/bin:/usr/local/fabric/system:/usr/java/default/bin
Aug 01 04:05:26 00:11:43:CD:FA:A8 FabricWorkerManager[I:10002] FabricWorkerManager OS ld_library_path: /usr/java/default/jre/lib/i386:/usr/java/default/jre/lib/i386/client:/usr/local/fabric/system:.
Aug 01 04:05:26 00:11:43:CD:FA:A8 FabricWorkerManager[I:10003] FabricWorkerManager Home: /usr/local/fabric/system
Aug 01 04:05:26 00:11:43:CD:FA:A8 RegionService[I:8000] Location: 10.10.10.61 Region: 239.255.0.75:32000 Services starting
Aug 01 04:09:22 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:48 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:07:59 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:34 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:05:34 00:11:43:CD:FA:A8 RegionService[I:8010] Location: 10.10.10.61 RegionLeaderRegion: Creating Region Leader for region
Aug 01 04:05:34 00:11:43:CD:FA:A8 RegionService[I:8008] Location: 10.10.10.61 RegionLeaderRegion: Creating Senior Region Leader for leader region
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:2006] App [spring_hello_world_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:2006] App [hello_world_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:2006] App [monte_carlo_pi_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:2006] App [extended_hello_world_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:4500] Submitting start command to fabric_task_service.extended_hello_world_java_app
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[E:4502] Error received while submitting start command to fabric_task_service.extended_hello_world_java_app : The service did not respond to the start or control request in a timely fashion.
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:4500] Submitting start command to fabric_task_service.hello_world_java_app
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[E:4502] Error received while submitting start command to fabric_task_service.hello_world_java_app : The service did not respond to the start or control request in a timely fashion.
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:4500] Submitting start command to fabric_task_service.monte_carlo_pi_java_app
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[E:4502] Error received while submitting start command to fabric_task_service.monte_carlo_pi_java_app : The service did not respond to the start or control request in a timely fashion.
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:4500] Submitting start command to fabric_task_service.spring_hello_world_app
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[E:4502] Error received while submitting start command to fabric_task_service.spring_hello_world_app : The service did not respond to the start or control request in a timely fashion.
Aug 01 04:05:34 00:11:43:CD:FA:A8 TaskServiceManager[I:4504] Done Submitting start commands to application services
Aug 01 04:05:34 00:11:43:CD:FA:A8 ProcessService[I:2006] App [spring_hello_world_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 ProcessService[I:2006] App [hello_world_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 ProcessService[I:2006] App [monte_carlo_pi_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 ProcessService[I:2006] App [extended_hello_world_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'extended_hello_world_java_app.hello_world_five_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'extended_hello_world_java_app.hello_world_four_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'extended_hello_world_java_app.hello_world_one_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'extended_hello_world_java_app.hello_world_three_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'extended_hello_world_java_app.hello_world_two_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'hello_world_java_app.hello_world_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3010] Process Flow: 'monte_carlo_pi_java_app.monte_carlo_pi_flow' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:3037] App(spring_hello_world_app) Component method: 'hello_world_component.greet(java.lang.String)' loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 FabricProcess[I:1000] Process Service started on port 16002
Aug 01 04:05:34 00:11:43:CD:FA:A8 RequestService[I:2006] App [spring_hello_world_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 RequestService[I:2006] App [hello_world_java_app] definition loaded
Aug 01 04:05:34 00:11:43:CD:FA:A8 RequestService[I:2006] App [monte_carlo_pi_java_app] definition loaded
Aug 01 04:05:35 00:11:43:CD:FA:A8 RequestService[I:2006] App [extended_hello_world_java_app] definition loaded
Aug 01 04:05:35 00:11:43:CD:FA:A8 FabricRequest[I:1000] Request Service started on port(s) 16000
Aug 01 04:05:36 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.47:40964
Aug 01 06:01:24 00:1E:C9:B2:4F:D1 FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 04:05:36 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.63:32782
Aug 01 04:05:36 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.64:32776
Aug 01 04:09:30 00:11:43:CE:0C:D4 FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 03:06:56 00:11:43:CE:0D:CE FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 04:05:36 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.62:32931
Aug 01 04:05:36 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.65:32777
Aug 01 04:08:07 00:11:43:CE:0E:0C FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 03:06:42 00:11:43:CE:0D:9A FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 06:01:24 00:1E:C9:B2:4F:D1 FabricKeeper[I:7124] Transferring file: libfabric.so, version 3.8.1.3
Aug 01 04:05:37 00:11:43:CD:FA:A8 FabricKeeper[I:7110] got new file (upload/libfabric.so) - running postprocessing
Aug 01 04:05:40 00:11:43:CD:FA:A8 FabricKeeper[I:7108] Spreading file: published libfabric.so
Aug 01 04:05:40 00:11:43:CD:FA:A8 FabricKeeper[I:7108] Spreading file: libfabric.so (3.8.1.3)
Aug 01 04:05:42 00:11:43:CD:FA:A8 RegionService[I:8011] Location: 10.10.10.61 Region: 239.255.0.75:32000 Heartbeat size: 1
Aug 01 04:05:45 00:11:43:CD:FA:A8 RegionService[I:8011] Location: 10.10.10.61 RegionLeaderRegion: Heartbeat size: 1
Aug 01 03:07:05 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:09:39 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:51 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:08:16 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:05:50 00:11:43:CD:FA:A8 RegionService[I:8011] Location: 10.10.10.61 Region: 239.255.0.75:32000 Heartbeat size: 1
Aug 01 04:05:51 00:11:43:CD:FA:A8 FabricKeeper[I:7113] Manifest Exchange: responding to 10.10.10.47:40985
Aug 01 06:01:39 00:1E:C9:B2:4F:D1 FabricKeeper[I:7110] exchanging manifests with leader: 10.10.10.61:44142
Aug 01 06:01:39 00:1E:C9:B2:4F:D1 FabricKeeper[I:7124] Transferring file: libfabric.so, version 3.8.1.3
Aug 01 04:05:52 00:11:43:CD:FA:A8 FabricKeeper[I:7110] got new file (upload/libfabric.so) - running postprocessing
Aug 01 04:09:46 00:11:43:CE:0C:D4 FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:07:12 00:11:43:CE:0D:CE FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 03:06:58 00:11:43:CE:0D:9A FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:08:23 00:11:43:CE:0E:0C FabricKeeper[I:7126] MFTP Transfer succeeded for: libfabric.so (3.8.1.3)
Aug 01 04:05:56 00:11:43:CD:FA:A8 RegionService[I:8011] Location: 10.10.10.61 RegionLeaderRegion: Heartbeat size: 1
Aug 01 04:05:58 00:11:43:CD:FA:A8 RegionService[I:8011] Location: 10.10.10.61 Region: 239.255.0.75:32000 Heartbeat size: 1
Thanks for gathering that information. I think there are two issues here. I will contact you directly via email. I will post the resolution here afterward.
The end result of this issue was that SE Linux was turned on. By default, it is on for CentOS 5.
After doing the following to disable SE Linux, everything worked:
change "enforcing" to "disabled" in /etc/selinux/config and reboot