1. Geofest portlet development ( status: working )

The goal is to build a new Geofest portlet for Gridsphere by using MVC model. I will not improve the old Jetspeed stuff.

Step 1. Create view layer
There are some application management stuff in the original GeoFEST portlets source code for Jetspeed. In the new portlet, we should not put any logic code in view layer. We want to get a clear, simple view layer by seperate old jsp source code. Now I am rewriting the jsp pages to JSF pages.

Step 2. Create model layer
I need some time to understand all of the original codes, then I will
accelerate this step work. I will separate all the logic codes from
jsp pages to model bean layer. I will start this step after I finished the first step.

Step 3. Create context manager
The old Geofest portlet using a service called the Contextmanager to preserve the bean information. That way is a little obscure. How can we make a better way?
Maybe we can use database plus hibernate.
db4o: read and write objects to/from a database
http://www.db4o.com/community/
And I have tried the db4o, it is a nice tool.
when I have a clear idea, I will rewrite the context manger.
This step should be the final work.

2. Created a simple map-movie demo (Status: finished)

The Map-move portlet URL is here:
http://gf1.ucs.indiana.edu:13080/gridsphere/gridsphere
USER: testuser
password: testuser
The Map-movie standalone JSF URL:
http://gf1.ucs.indiana.edu:13080/TestMovie-portlet/allstations.jsp

I added the new movie URL and make a portlet war file.
You can download the war file from:
http://gf2.ucs.indiana.edu:23080/sensorgrid_test/TestMovie-portlet.war
* Notes on Google Maps: this portlet does not work with IE but Firefox
is OK. Also, as always, the google map signature on the portlet must
be specific to your host name. you can get the instruction from google map api:
http://www.google.com/apis/maps/documentation/
You just need change the API key in "allstations.jsp" file.

1. for the perfermance test of Sensorgrid(status: finished)

test 4– Multiple Broker Test (status: finished)
You can find the test structure in the previous report.
Noticed:
We used two benchmark clients to record the time stamp.
Benchmark client 1 connected with Narada broker server1.
Benchmark client 2 connected with Narada broker server2.
The result benchmark client 1 recorded is picuture 1.
The result benchmark client 1 recorded is picuture 2.
Follow the pictures both 1 and 2:
Before the 1th hour, I add 1400 subscribers to system each 10 seconds.After the 1th hour the system works with 1400 subscribers for another 24 hours. The blue line is the 1th hour boundary.
This test shows that the system with multi-broker scales up to 1400 clients with an acceptable transfer delay. The delay time between two brokers is quite small, just 5 ms.
This test also shows our framework can exceed 1000 subscribers with multi-broker support.
If we want to support huge number clients , we just need add more brokers to a distribute network.

Note:
If NB received this error "TCPReceiverThread: MAJOR ERROR:
Received Data with garbled transport header", the NB will be down.
That error is definitely fatal,maybe it caused by the I/O, since 700 readers
are trying to read the same file, it is possible a garbled data.
I also got some failed tests,maybe those failed test is crushed by server errors or I/O errors.
But we know the minimal thing is that the NB server will be down when
it received this error "TCPReceiverThread: MAJOR ERROR:Received Data
with garbled transport header".
If NB can handle those errors, our sensorgrid framework will be more stable.

The tests enviroment is here:
The perfermance test of Sensorgrid.pdf

The test 1 result is here:
test1_result.tar.gz

The test 2 result is here:
test2_result.tar.gz

The test 3 result is here:
test3_result1.tar.gz
test3_result2.tar.gz

The test 4 result is here:
test4_result.tar.gz

By those 4 tests, we can get:
Our framework is quite strong with these characteristics as follows:
1- stable for continuous operation
2- Has ability to support multiple publishers
3- Has ability to support multiple clients
4- The message delivery times is quite small
5- Has ability to preserve the order of incoming messages


Future work:
GeoFEST stuff(discuss it with Marlon and Yili this Friday )
STfilter portlets and services(discuss it with Marlon this Friday)

1. for the perfermance test of Sensorgrid(status: working)
retest 1 – Measuring the performance of the Basic System (status: finished)
I started a retest for the basic system perfermance test, I got a better result.There is not any jumps with the new result.The result shows that the transfer time is stable for average around 6ms.All the tests for the basic sytem show that the continuous operation does notdegrade the system performance.

retest 3– Finding the maximum number of clients single broker can support (status: finished)
I started a retest for the test3, and I got a better result.
Follow the picture:
Before the 5th hour, we add 100 subscribers to system each 30 minutes.After the 5th hour the system works with 1000 subscribers for another 19 hours. The blue line is the 5th hour boundary.
Why we just use 1000 subscribers for the test?
The default maximum number of FD(file descriptors) redhat linux OS allowed is 1024. That means one process can not exceed 1024 TCP connections. We can not add more subscribers to the system with only one NaradaBroker server.
This test shows that the system with a single broker scales up to a thousand clients with an acceptable transfer delay.

test 4– Multiple Broker Test (status: doing)
The first three tests show that our framework is stable for the continuous operation and scales up to the maximum number of file descriptors allowed by the server’s operating system. To support many more clients or producers, we will create a distributed NaradaBrokering broker network. So we start a new test to prove this ability.
the test model is as follow:

I have not finished this test, A middle result is as follow:

2. for the process monitoring stuff (status: working)

ProcessMonitor is updated to 0.2 version.
New features:
Added PID list and UID list
Process Monitor will watch those lists, if some process exit, then send email to specified user.
Refined the calculate-load function.
The new version URL:
ProcessMonitor.0.2.tar.gz


Future work:
try to finish the test 4 and update the real-time Rdahmm map URL service.

# posted by Zhigang Qi : 6:56 PM  0 comments

Steps for Connecting two or more NaradaBrokering Nodes (From Galip, I just fix some errors)
We have two brokers,
Broker A running on tcp://gf3:5544
(change AssignedAddress=false to AssignedAddress=true The first Broker node should assigns its own address, that mean "AssignedAddress" parameter should be true.)
Broker B running on tcp://gf1:5545
(AssignedAddress=false)
1. Go to BROKER_B_HOME/config open BrokerConfiguration.txt, change AssignedAddress=false Save and Exit.
2. Start Broker A
3. Start Broker B
4. For Broker B, run ./brokerInteract.sh Type h to see usage
Type [c gf3.ucs.indiana.edu 5544 t] [t for tcp nio for niotcp]
After the connection is done copy the link address printed at the end of the message and type [na linkaddress 0]
i.e. [na tcp://gf3.ucs.indiana.edu:5544 0]
[note](from NaradaBroker User documentation):
Other brokers can contact this broker to help set up the distributed network. Please note that the first broker on a NB broker network assigns its own address.If the AssignedAddress parameter is set to false the broker does not assign itself an address and is ready to be part of a distributed broker network.

1. for the perfermance test of Sensorgrid(status: working)

The main goal of the tests is to find out if our filter architecture is scalable for use with large numbers of real time data providers and clients.

test 1 – Measuring the performance of the Basic System (status: finished)
The first test is to run the basic system for 24 hours and take the measurements. We test the basic system with 87845 messages.
Follow the picture, we find:
Continuous operation will not degrade the system performance and the messages will be delivered in the incoming order. The average message delivery time is 5.
By this basic system test, we can prove that our filter architecture is consistent in performance or behavior for a long time running.
test 2– Finding the maximum number of GPS networks for a single broker(status: finished)
In this group of tests, we will increase the number of RYO publisher to single NB broker.
We run the tests for 48 hours and take the measurements.
After I added 1000 Publishers to NB server, the NaradaBroker with 50% CPU usage still work fine, the delay time is only 5-6 ms.
This test shows that our filter architecture is scalable for use with large numbers of real time data providers.
test 3– Finding the maximum number of clients a single broker can support (status: doing)
In this test, I increased the number of Simple Filters to simulate the real-time data clients for only one publisher to NaradaBroker.
I have not finished this test, but I got some interesting results.
I will discuss this result with Galip.
Look the result picture:
The NaradaBroker work fine with few than 500 clients, the delay time is bellow 10 ms.
After I added 1000 clients to NB server, the NaradaBroker work fine with only 10% CPU usage , but the delay time is up to 40 ms.
We can find a suddenly jump in this picture. this behavior is very strange.
so we need think about this jump, this is not a natural increase, How can we explain it?
I will do a retest for the test 3 to see if this behavior was made by some strange server error.
2. for the process monitoring stuff (status: working)
ProcessMonitor is a test tool for testing SensorGrid filters service and narada service.
Description:
ProcessMonitor is a daemon which checks the current load of a server and take user-defined action to processes which consume too much CPU time. The owner of a process is informed by email. This tool is writed by C++.
I want to add a new function to this tool.
I am working on this test tool, I will update this test tool this Friday.
The preversion URL:
ProcessMonitor.0.1.tar.gz
3. for the portal development work (status: working)
I have tried to make simple JSF-Bean example by using gridsphere portal framework last week.
I am tring to add the real time Rdahmm map service to the Quakesim2 project.

4. for the real-time Rdahmm map service (status: working)
I have updated the real-time Rdahmm map service.
Follow Michael's request, the URL of Realtime-rdahmm pictures was changed.
I also made this into a real web service. Input would be the station name.
Output would be the URL list of pictures.
# RdahmmPlotURL (wsdl)
* QueryRdahmmURL -- for Rdahmm plot
* QueryPlainURL -- for plain plot
If the service can not find the queried station from the file server,
then return a error code:404.
The WSDL URL is:
http://gf2.ucs.indiana.edu:23080/axis/services/RdahmmPlotURL?wsdl