----- Original Message ----- > Updated Branches: > refs/heads/master 8d8507f3f -> b36a7e120 > > > document update for hipes and buffer_upload plugins > > > Project: http://git-wip-us.apache.org/repos/asf/trafficserver/repo > Commit: http://git-wip-us.apache.org/repos/asf/trafficserver/commit/b36a7e12 > Tree: http://git-wip-us.apache.org/repos/asf/trafficserver/tree/b36a7e12 > Diff: http://git-wip-us.apache.org/repos/asf/trafficserver/diff/b36a7e12 > > Branch: refs/heads/master > Commit: b36a7e120e4b9c216fe2f13a97015a508911e0a4 > Parents: 8d8507f > Author: Kit Chan <kic...@apache.org> > Authored: Sat Dec 28 09:06:04 2013 -0800 > Committer: Kit Chan <kic...@apache.org> > Committed: Sat Dec 28 09:06:04 2013 -0800 > > ---------------------------------------------------------------------- > doc/reference/plugins/buffer_upload.en.rst | 93 +++++++++++-------------- > doc/reference/plugins/hipes.en.rst | 22 ++++-- > 2 files changed, 58 insertions(+), 57 deletions(-) > ---------------------------------------------------------------------- > > > http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b36a7e12/doc/reference/plugins/buffer_upload.en.rst > ---------------------------------------------------------------------- > diff --git a/doc/reference/plugins/buffer_upload.en.rst > b/doc/reference/plugins/buffer_upload.en.rst > index ab5f209..022d61e 100644 > --- a/doc/reference/plugins/buffer_upload.en.rst > +++ b/doc/reference/plugins/buffer_upload.en.rst > @@ -18,72 +18,63 @@ Buffer Upload Plugin > specific language governing permissions and limitations > under the License. > > +The Buffer Upload plugin offers the following features > > -.. XXX Discribe what the heck this plugin actually does. > +Installation > +============ > > -Upload proxy specs for phase I: > +Configuration can be explicitly specified as a parameter in > ``plugin.config`` :: > > -1. Memory buffering (buffer the entire POST data in IOBuffer before > - connecting to OS) 1.1. Memory buffer size is configured with > - "mem_buffer_size" in config file. Default and minimum value is 32K > - You can increase it in the config file. If the size of a request is > - larger than the "mem_buffer_size" value specifiied in the config > - file, then the upload proxy feature will be disabled for this > - particular request > + buffer_upload.so /FOOBAR/upload.conf > > -2. Disk buffering (buffer the entire POST data on disk before connecting > - to OS) 2.1. Use disk async IO. This involved some changes in ATS core > - . new APIs wrapping around ink_aio_read() and ink_aio_write() . > - change to distinguish between api call's AIO and cache's AIO . > - guarantee api call's AIO only involves certain amount of threads . > - the number of threads is configurable in plugin's config file > - (default is 4) > +Memory buffering (buffer the entire POST data in IOBuffer before connecting > to OS) > +================================================================================== > > -3. > +Memory buffer size is configured with "mem_buffer_size" in config file. > Default and minimum value is 32K. You can
This should default to our POST max size, which currently, IIRC, is 64K. > +increase it in the config file. If the size of a request is larger than the > "mem_buffer_size" value specifiied in the > +config file, then the upload proxy feature will be disabled for this > particular request > > - 2. Directories and files generated on disk . base directory: > - FOOBAR/var/buffer_upload_tmp/ (configurable in config file) . > - number of subdirectories: 64 (configurable in config file) . > - filename are randomly generated . files will be removed when the > - entire data have been sent out to OS . remove dangling files (left > - on disk due to transaction interruption or traffic server crash) > - at startup time > +Disk buffering (buffer the entire POST data on disk before connecting to OS) > +============================================================================ > > -4. > +1. Disk async IO is used. AIO api call only involves certain amount of > threads. The number of threads is configurable in > +plugin's config file (default is 4) > > - 3. Default chunk size when reading from disk: 16K, configurable in > - config file > +2. Directories and files are generated on disk . Base directory is > /FOOBAR/var/buffer_upload_tmp/ (configurable in > +config file). Number of subdirectories is 64 (configurable in config file). > Filename are randomly generated. Files will > +be removed when the entire data have been sent out to OS . At startup time, > dangling files are removed (left on disk due > +to transaction interruption or traffic server crash) > > -5. Default buffering mode: disk aio buffering mode 3.1. to turn off disk > - buffering, add a "use_disk_buffer 0" line in config file > +3. Default chunk size when reading from disk is 16K, configurable in config > file > > -6. Trigger POST buffering on certain URLs 4.1. certain URLs will be > - provided in a plain text file (one URL each line) 4.2. specify > - filename in config file by "url_list_file" 4.3. max length of each > - URL: 4096 (configurable in config file) 4.4. use exact match, don't > - support regex for now > +Trigger POST buffering on certain URLs > +====================================== > > -7. URL conversion for Mail's specific URL format 5.1. for now check if > - the "host" part in the URL is same as the proxy server name, then > - will do this conversion 5.2. To turn on URL conversion feature, set > - "convert_url 1" in config file > +1. Certain URLs will be provided in a plain text file (one URL each line) > +2. Specify filename in config file by "url_list_file" > +3. max length of each URL is 4096 (configurable in config file) > +4. use exact match, don't support regex for now > > -8. All request headers inlcuding cookies plus the entire POST data will > - be buffered (either in memory or on disk) > +Other Features > +============== > > -9. Config file can be explicitly sepcified as a parameter in command > - line (in plugin.config file) > +1. Default buffering mode is disk aio buffering mode. To turn off disk > buffering, add a "use_disk_buffer 0" line in > +config file > > -a sample config file: > +2. All request headers inlcuding cookies plus the entire POST data will be > buffered (either in memory or on disk) > > -use_disk_buffer 1 convert_url 1 chunk_size 1024 url_list_file > -/tmp/url_list.conf max_url_length 10000 base_dir /tmp/test1 > -subdir_num 100 thread_num 10 mem_buffer_size 40000 > +Configuration File > +================== > > -default config file: FOOBAR/etc/upload.conf > +sample config file :: > > -default config values: use_disk_buffer 1 convert_url 0 chunk_size > -16384 url_list_file none max_url_length 4096 base_dir > -FOOBAR/var/buffer_upload_tmp subdir_num 64 thread_num 4 > -mem_buffer_size 32768 > + use_disk_buffer 1 > + convert_url 1 > + chunk_size 1024 > + url_list_file /tmp/url_list.conf > + max_url_length 10000 > + base_dir /tmp/test1 > + subdir_num 100 > + thread_num 10 > + mem_buffer_size 40000 > > > http://git-wip-us.apache.org/repos/asf/trafficserver/blob/b36a7e12/doc/reference/plugins/hipes.en.rst > ---------------------------------------------------------------------- > diff --git a/doc/reference/plugins/hipes.en.rst > b/doc/reference/plugins/hipes.en.rst > index e74a58a..20c847b 100644 > --- a/doc/reference/plugins/hipes.en.rst > +++ b/doc/reference/plugins/hipes.en.rst > @@ -62,7 +62,8 @@ only apply to the connection to the specific "service" > host. > Notes on HIPES > ============== > > -HTTP Pipes (aka HIPES and pronounced "Hippies") allows data services to be > pipelined together, as illustrated by the example below. > +HTTP Pipes (aka HIPES and pronounced "Hippies") allows data services to be > pipelined together, as illustrated by the > +example below. > > Example > ======= > @@ -90,25 +91,34 @@ to this URL :: > > > http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt > > -5. In this example, the service ``svc_case.php`` fetches and transforms the > response of ``http://127.0.0.1/test.txt`` (which ATS proxies the request to > a local file) to upper case. And subsequently the service > ``svc_reverse.php`` receives the response and reverse the order before the > response is sent back to the client by ATS. > +5. In this example, the service ``svc_case.php`` fetches and transforms the > response of ``http://127.0.0.1/test.txt`` > +(which ATS proxies the request to a local file) to upper case. And > subsequently the service ``svc_reverse.php`` receives > +the response and reverse the order before the response is sent back to the > client by ATS. > > Notes on reducing traffic > ========================= > > -There can be significant overhead using HIPES because the data can traverse > through ATS many times. Caching can be important to reduce traffic to > services/through ATS and can be achieved via a proper ``Cache-Control`` > header returned by the services. Another way to reduce traffic through ATS > is to have ATS to return 302 redirects to url for the requests made by > service, instead of proxying the requests to that url. However, the service > must then be able to follow the redirect. The down side is that we cannot > use ATS to cache intermediate results. Below is an example of using > redirect. > +There can be significant overhead using HIPES because the data can traverse > through ATS many times. Caching can be > +important to reduce traffic to services/through ATS and can be achieved via > a proper ``Cache-Control`` header returned > +by the services. Another way to reduce traffic through ATS is to have ATS to > return 302 redirects to url for the > +requests made by service, instead of proxying the requests to that url. > However, the service must then be able to follow > +the redirect. The down side is that we cannot use ATS to cache intermediate > results. Below is an example of using > +redirect. > > Modification to above example to reduce traffic using redirect > ============================================================== > > -1. The service ``svc_reverse.php`` is modified to add a header of > ``X-HIPES-Redirect: 2`` to the request made against ``url``. > +1. The service ``svc_reverse.php`` is modified to add a header of > ``X-HIPES-Redirect: 2`` to the request made against > +``url``. > > 2. HIPES plugin will instruct ATS to return a redirect response to this url > :: > > > http://127.0.0.1:8080/svc_case.php?case=u&url=http%3A%2F%2F127.0.0.1%2Ftest.txt > > -for the following request > +for the following request :: > > http://127.0.0.1/svc_case;case=u/test.txt > > -3. The service ``svc_reverse.php`` is also modified to follow the redirect. > Thus the response of the service of ``svc_case.php`` will not pass through > ATS and will pass to ``svc_reverse.php`` service instead. > +3. The service ``svc_reverse.php`` is also modified to follow the redirect. > Thus the response of the service of > +``svc_case.php`` will not pass through ATS and will pass to > ``svc_reverse.php`` service instead. > > > -- Igor Galić Tel: +43 (0) 664 886 22 883 Mail: i.ga...@brainsware.org URL: http://brainsware.org/ GPG: 8716 7A9F 989B ABD5 100F 4008 F266 55D6 2998 1641
