Monday, March 23, 2009
Daily RDAHMM portlet modified corresponding to the video service
We have modified the daily RDAHMM portlet so that it now links to a video file made by the daily RDAHMM video service, which will always be in mp4 format.
Monday, March 16, 2009
Daily RDAHMM Video Web Service Completed
And here's the design document:
To make daily RDAHMM movies with h264 video format, which is the most portable format across Linux, Windows and Mac platforms, we decide to create a DailyRDAHMMVideoService web service sitting on a machine where everything is OK for making h264 videos, and answering the requests from DailyRDAHMMService, and returning links to videos newly created.
Pre-requisite:
mencoder installed with support for x264 options
mp4BOX installed
So far the only machine that satisfies these requirements is Xiaoming's machine with IP address 156.56.104.161.
Requirements for DailyRDAHMMVideoService web service:
We call DailyRDAHMMVideoService the "video service" for short. This service is supposed to support the creation of daily RDAHMM movies for different data sources (SOPAC, JPL, etc.), and different pre-process treatment methods to input data (with vs. without missing data treatment). Given a data source, a treatment method, a URL to the analysis result XML file of that data souce, and a pair of start and end date, the video service will create a movie for the period between the start and end date, according to the result XML file specified in that URL, and return a URL to the created video. More specificly, the video service provides a web service method "String makeVideo(String dataSource, String preProcTreat, String resultURL, String startDate, String endDate)", which makes the requested video, and returns the URL string of the video.
Design Issues:
1. Directory seperation. Since we want to support different data sets and input treatment methods, the analysis results and corresponding map pictures for these different parameters will be different, so we need to separate the directories for different parameter combinations. Let denote the top directory for storing the pictures and temporary video files, we use [picDir]/[dataSouce]/[preProcTreat] for different parameters.
2. Quick response. To respond quickly to the client, we apply asynchronous process to its request. First, the video service will check if the parameters are valid, e.g., whether the given data source and treatment methods are supported. If not, a URL pointing to another method of the video service, "String getErrorMsg(int msgNum)" will be returned to the client, so that when the client opens that URL, a proper error message will be returned. If the parameters are valid, a URL will be constructed immediately according to these parameters and returned to the client, before the video is actually created. That is, we assume that there will be no errors in the process of creating the video. After enough debugging, we think this will not cause very serious problems: anyway, putting an invalid URL on the portal in rare cases will not be the end of the world.
3. Requests buffering. In order to answer multiple requests asynchronously in a short time, we buffer these results in a list, and use a separate thread to process them. This separate thread is sleeping when there is no request to process, and will be waken up for new requests.
4. Avoiding redundant work. To avoid creating video frames for any period for multiple times, we save the start date (which will always be 1994-01-01) and end date of the last video created. Upon every new request, the video service will check its start and end date. If these dates falls in between the start and end date of the last video, we just return the URL for the last video. A new video will be created only if the end date is later than the end date of the last video, since all starting dates are supposed to be no earlier than 1994-01-01. Besides, if the month number of the given end date minus 2 is later than the month number of the historical video, the historical video is updated and extended to the month according to the given month number minus 2. For example, if the end date of the historical video is 2008-07-31, and the end date of the requested video is 2008-11-22, then the historical video will be updated to 2008-09-30 first, and then the new video is created based on the new historical video.
To make daily RDAHMM movies with h264 video format, which is the most portable format across Linux, Windows and Mac platforms, we decide to create a DailyRDAHMMVideoService web service sitting on a machine where everything is OK for making h264 videos, and answering the requests from DailyRDAHMMService, and returning links to videos newly created.
Pre-requisite:
mencoder installed with support for x264 options
mp4BOX installed
So far the only machine that satisfies these requirements is Xiaoming's machine with IP address 156.56.104.161.
Requirements for DailyRDAHMMVideoService web service:
We call DailyRDAHMMVideoService the "video service" for short. This service is supposed to support the creation of daily RDAHMM movies for different data sources (SOPAC, JPL, etc.), and different pre-process treatment methods to input data (with vs. without missing data treatment). Given a data source, a treatment method, a URL to the analysis result XML file of that data souce, and a pair of start and end date, the video service will create a movie for the period between the start and end date, according to the result XML file specified in that URL, and return a URL to the created video. More specificly, the video service provides a web service method "String makeVideo(String dataSource, String preProcTreat, String resultURL, String startDate, String endDate)", which makes the requested video, and returns the URL string of the video.
Design Issues:
1. Directory seperation. Since we want to support different data sets and input treatment methods, the analysis results and corresponding map pictures for these different parameters will be different, so we need to separate the directories for different parameter combinations. Let denote the top directory for storing the pictures and temporary video files, we use [picDir]/[dataSouce]/[preProcTreat] for different parameters.
2. Quick response. To respond quickly to the client, we apply asynchronous process to its request. First, the video service will check if the parameters are valid, e.g., whether the given data source and treatment methods are supported. If not, a URL pointing to another method of the video service, "String getErrorMsg(int msgNum)" will be returned to the client, so that when the client opens that URL, a proper error message will be returned. If the parameters are valid, a URL will be constructed immediately according to these parameters and returned to the client, before the video is actually created. That is, we assume that there will be no errors in the process of creating the video. After enough debugging, we think this will not cause very serious problems: anyway, putting an invalid URL on the portal in rare cases will not be the end of the world.
3. Requests buffering. In order to answer multiple requests asynchronously in a short time, we buffer these results in a list, and use a separate thread to process them. This separate thread is sleeping when there is no request to process, and will be waken up for new requests.
4. Avoiding redundant work. To avoid creating video frames for any period for multiple times, we save the start date (which will always be 1994-01-01) and end date of the last video created. Upon every new request, the video service will check its start and end date. If these dates falls in between the start and end date of the last video, we just return the URL for the last video. A new video will be created only if the end date is later than the end date of the last video, since all starting dates are supposed to be no earlier than 1994-01-01. Besides, if the month number of the given end date minus 2 is later than the month number of the historical video, the historical video is updated and extended to the month according to the given month number minus 2. For example, if the end date of the historical video is 2008-07-31, and the end date of the requested video is 2008-11-22, then the historical video will be updated to 2008-09-30 first, and then the new video is created based on the new historical video.
Wednesday, February 11, 2009
Compilation problem with x264 enabled mencoder
In order to make a daily RDAHMM movie with fps=5 that is compatible to every platform and player, we tried to encode the video with several different types of formats, and found that the only qualified format is h264. However, in order to make mencoder work with h264, we need to recompile it with x264 option, as well as the support of x264 package. We found it very hard to make sure the compilation can complete on any server, because of lack of x264 support on the machine, or because the wrong version of x264.
So we are now trying to separate the video making function from the Daily RDAHMM Service, so that it can just be deployed to a server where everything is OK, and answer requests from Daily RDAHMM Service from everywhere. This way we also need only one copy of all the map pictures and historical video, and the update of the historical video will also be easier to manage.
So we are now trying to separate the video making function from the Daily RDAHMM Service, so that it can just be deployed to a server where everything is OK, and answer requests from Daily RDAHMM Service from everywhere. This way we also need only one copy of all the map pictures and historical video, and the update of the historical video will also be easier to manage.
Daily RDAHMM Result Service added to svn control
The Daily RDAHMM Service has been added to the code depository of QuakeSim2. A new directory DailyRDAHMMResultService is added to the ExecutionServices directory, and the service can be built and deployed with maven commands from the top QuakeSim2 directory.
Since Daily RDAHMM Result Service is deployed with Axis2, we also added a axis2.tar.gz under the portal_deploy directory, which will be deployed after tomcat is deployed. The Daily RDAHMM Result Service is deployed under the "services" directory of axis2.
For sake of consistency with previous deployment configurations, the Daily RDAHMM Result Service will also be built with maven scripts in pom.xml, and a directory will be copied to the webapps directory of tomcat. But this is not necessary: this directory will not be the running service. We might be able to deploy all other services to axis2 in the future, so that we don't need to copy them to "webapps" directly any more.
Since Daily RDAHMM Result Service is deployed with Axis2, we also added a axis2.tar.gz under the portal_deploy directory, which will be deployed after tomcat is deployed. The Daily RDAHMM Result Service is deployed under the "services" directory of axis2.
For sake of consistency with previous deployment configurations, the Daily RDAHMM Result Service will also be built with maven scripts in pom.xml, and a directory will be copied to the webapps directory of tomcat. But this is not necessary: this directory will not be the running service. We might be able to deploy all other services to axis2 in the future, so that we don't need to copy them to "webapps" directly any more.
Tuesday, December 2, 2008
Test DailyRdahmm Result Service set up
We have set up a test web service at http://156.56.104.161:8080/axis2/services/DailyRdahmmResultService?wsdl to calculate markers' colors for a given date. Also added is a "please wait" window to the portlet while the page is waiting for results from the service. Calling the web service to calculate the colors is actually faster than using managed bean, because now we don't have to refresh the page. The test page is located at http://156.56.104.161:8080/gridsphere/gridsphere, log in with empty user name and password.
We'll update this to gwll and svn after performance measure is done.
We'll update this to gwll and svn after performance measure is done.
Saturday, November 1, 2008
Daily RDAHMM video updated
Because of the changes of our treatment to missing data sections, and correction of several bugs, the historical results of Daily RDAHMM are also different from previously. So we just made a new movie for the whole time since 1994 based on the new results. The new video can be accessed from the portal at http://gw11.quarry.iu.teragrid.org:8080/gridsphere/gridsphere
Modifications to DailyRDAHMM missing data treatment
We did the following modifications to the treatment of missing data in DailyRDAHMM:
1. For the input data, if the dates used for getting the input is 1994-01-01 to 2006-09-30, and we only get data for 1995-10-03 to 2006-8-22, then we will only fill those time sections between 1995-10-03 and 2006-8-22, and leave the gaps before and after as they are, in order to avoid the distortion to results;
2. For the big flat file that contains input for all stations, we use NaNs to fill all missing lines.
1. For the input data, if the dates used for getting the input is 1994-01-01 to 2006-09-30, and we only get data for 1995-10-03 to 2006-8-22, then we will only fill those time sections between 1995-10-03 and 2006-8-22, and leave the gaps before and after as they are, in order to avoid the distortion to results;
2. For the big flat file that contains input for all stations, we use NaNs to fill all missing lines.
Subscribe to:
Posts (Atom)