I have got ATS working end to end. Please find below edge and mid configurations. In this configuration the Host header is set to parent cache (i.e. vcache-mid-1.cdn.lab:31695) and because of that, the edge can contact the parent.
But in this configuration, the edge is not aware of the originserver where it might need to contact directly in case of parent is down. It would be great to somehow rewrite the Host header to be set with the selected parent cache, right before the request is being forwarded to the parent cache, where in that case the edge would remap the request to the originserver as before. Any advise, would be helpful. I am using ATS v8.0.8 Thanks, - Avi Edge ATS: ========= remap.config ------------ map http://vcache-edge-1.cdn.lab:31695/myCI/ http://{cache} map http://vcache-edge-1.cdn.lab:31695/ http://vcache-mid-1.cdn.lab:31695/ reverse_map http://vcache-mid-1.cdn.lab:31695/ http://vcache-edge-1.cdn.lab:31695/ parent.config ------------- dest_domain=. parent="vcache-mid-1.cdn.lab:31695" Mid ATS: ======== remap.config ------------ map http://vcache-mid-1.cdn.lab:31695/myCI/ http://{cache} map http://vcache-mid-1.cdn.lab:31695/ http://originserver.cdn.lab:8080/ parent.config ------------- dest_domain=. parent="originserver.cdn.lab:8080" From: "Avi Weit" <w...@il.ibm.com> To: Jeremy Payne <jp557...@gmail.com> Cc: dev@trafficserver.apache.org Date: 26/06/2020 17:28 Subject: [EXTERNAL] Re: ATS forwarding requests to parent cache Hi, Thanks for your reply. Please find below my current configuration of Edge and Mid ATS, according to your suggestion (if I correctly understood). It seems to me that the remap.conf should contain originserver url in case of parent cache is down or unreachable. >From looking into the edge cache logs I can see that the Host header was not changed to contain destination hostname. It is still set to vcache-edge-1.cdn.lab:31695. My requirement is for the Host: header to be set to the hostname the request is directed to. If the request is forwarded to parent then it should be set to parent hostname (i.e. vcache-mid-1.cdn.lab:31695), if request directed to originserver (e.g. in case edge ATS detected that parent is down) then Host: needs to be set to originserver.cdn.lab:8080. I see the Host: header is set to vcache-edge-1.cdn.lab:31695 for all requests generated by edge ATS. Edge ATS ========= remap.conf ----------------- map https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_myCI_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Vkv0gQuKwAoHBYVpqwl6K8zop4R4pU9-G9fzDdaBxTY&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__-257Bcache-257D&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=c1Q-IJjfhBFSHuiL5yf0UAeWbjDyfF8Suu8HxECrQ40&e= map https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=yVEMXC3I0OKQzDjmERSqBndS8AAOX-odMcB7WLO0W6E&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Gg7mXXqmPOSH7eSmvMYjKvpUaAHlxxtKo6-htVc1dUI&e= reverse_map https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Gg7mXXqmPOSH7eSmvMYjKvpUaAHlxxtKo6-htVc1dUI&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=yVEMXC3I0OKQzDjmERSqBndS8AAOX-odMcB7WLO0W6E&e= parent.conf ----------------- dest_domain=. parent="vcache-mid-1.cdn.lab:31695" Parent ATS ========== remap.conf ----------------- map https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dmid-2D1.cdn.lab-3A31695_myCI_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=5noy8srrYwiAcT9ZxNCciYEoIOKtE2KDGHw2msLNgz4&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__-257Bcache-257D&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=c1Q-IJjfhBFSHuiL5yf0UAeWbjDyfF8Suu8HxECrQ40&e= map https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Gg7mXXqmPOSH7eSmvMYjKvpUaAHlxxtKo6-htVc1dUI&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Gg7mXXqmPOSH7eSmvMYjKvpUaAHlxxtKo6-htVc1dUI&e= parent.conf ----------------- dest_domain=. parent="originserver.cdn.lab:8080" Thanks, - Avi ========================================================================= From: Jeremy Payne <jp557...@gmail.com> Date: Thu, 25 Jun 2020 11:14:05 -0500 Message-ID: <CAEkTh3Fn05fKYUXUcjULbin=vkpmbj84jjtxipfphmqq_9e...@mail.gmail.com> Subject: Re: ATS forwarding requests to parent cache To: dev@trafficserver.apache.org Content-Type: text/plain; charset="UTF-8" ensure your parent cache will recognize requests for originserver.cdn.lab:8080 This below is the request to your parent cache. GET https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_player.html&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=KazpDdaeY10A52MJC1xVu-b8SW4yzRLgWvkw4x_mcg4&e= HTTP/1.1^M The HOST header in this request will not be honored. In this context, you have to account for the FQDN seen in the GET request. If you are not using direct_to_origin in the parent.config, then the easiest solution is to modify the origin definition in your edge remap.config. Or modify the remap on your parent to match the above FQDN seen in the GET request. From: Avi Weit/Haifa/IBM To: dev@trafficserver.apache.org Date: 25/06/2020 17:54 Subject: ATS forwarding requests to parent cache Hello, I have a use case where my ATS edge cache is deployed under kubernetes cluster A while its parent cache (i.e. ATS mid cache) is deployed inside a kubernetes cluster B. Cluster B has a front end of ingress controller (type nginx) that serves an ingress resource (Ingress kubernetes object) that routes requests destinated to vcache-mid-1.cdn.lab into the proper ATS deployment pod where traffic_manager is running. My ATS edge cache is able to receive user requests. However, it gets 404 (page not found) when forwarding the request to the patent cache. Please find below my remap.conf and parent.conf of the edge cache: remap.conf: ----------------- map https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_myCI_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Vkv0gQuKwAoHBYVpqwl6K8zop4R4pU9-G9fzDdaBxTY&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__-257Bcache-257D&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=c1Q-IJjfhBFSHuiL5yf0UAeWbjDyfF8Suu8HxECrQ40&e= map https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=yVEMXC3I0OKQzDjmERSqBndS8AAOX-odMcB7WLO0W6E&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Gg7mXXqmPOSH7eSmvMYjKvpUaAHlxxtKo6-htVc1dUI&e= reverse_map https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=Gg7mXXqmPOSH7eSmvMYjKvpUaAHlxxtKo6-htVc1dUI&e= https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=yVEMXC3I0OKQzDjmERSqBndS8AAOX-odMcB7WLO0W6E&e= parent.conf: ----------------- dest_domain=. parent="vcache-mid-1.cdn.lab:31695" I think this may be related to a more general question: how does ATS forward the request to a parent cache that is behind a loadbalancer i.e. nginx with a virtual host (e.g. of vcache-mid-1.cdn.lab) ? I think that if Host: header would contain parent hostname, then it would work. Please find below edge cache logs from traffic.out file: ... [Jun 25 14:19:27.052] Server {0x2b33b0df7700} DEBUG: (http_seq) [HttpSM::do_remap_request] Remapping request [Jun 25 14:19:27.052] Server {0x2b33b0df7700} DEBUG: (http_trans) START HttpTransact::EndRemapRequest [Jun 25 14:19:27.052] Server {0x2b33b0df7700} DEBUG: (http_trans) EndRemapRequest host is originserver.cdn.lab ... +++++++++ Incoming Request +++++++++ -- State Machine Id: 119 GET https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_player.html&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=KazpDdaeY10A52MJC1xVu-b8SW4yzRLgWvkw4x_mcg4&e= HTTP/1.1^M Host: vcache-edge-1.cdn.lab:31695^M X-Request-ID: 110fe9e2b1e8ba555c6b962d5b65b970^M X-Real-IP: 10.244.0.0^M X-Forwarded-For: 10.244.0.0^M X-Forwarded-Host: vcache-edge-1.cdn.lab:31695^M X-Forwarded-Port: 80^M X-Forwarded-Proto: http^M X-Scheme: http^M User-Agent: curl/7.58.0^M Accept: */*^M ^M ... [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (dns) received query vcache-mid-1.cdn.lab type = 1, timeout = 0 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [ink_cluster_time] local: 1593094767, highest_delta: 0, cluster: 1593094767 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [HttpTransact::PPDNSLookup] This was attempt 1 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [PPDNSLookup] DNS lookup for sm_id[119] successful IP: 172.15.0.170 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) client_ip_set = 0 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) inserted request header 'Client-ip: 10.244.1.23' [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [add_client_ip_to_outgoing_request] Appended connecting client's (10.244.1.23) to the X-Forwards header [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [build_request] request like cacheable and conditional headers removed [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [ink_cluster_time] local: 1593094767, highest_delta: 0, cluster: 1593094767 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) [build_request] request_sent_time: 1593094767 +++++++++ Proxy's Request +++++++++ -- State Machine Id: 119 GET https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_player.html&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=KazpDdaeY10A52MJC1xVu-b8SW4yzRLgWvkw4x_mcg4&e= HTTP/1.1^M Host: vcache-edge-1.cdn.lab:31695^M X-Request-ID: 110fe9e2b1e8ba555c6b962d5b65b970^M X-Real-IP: 10.244.0.0^M X-Forwarded-For: 10.244.0.0, 10.244.1.23^M X-Forwarded-Host: vcache-edge-1.cdn.lab:31695^M X-Forwarded-Port: 80^M X-Forwarded-Proto: http^M X-Scheme: http^M User-Agent: curl/7.58.0^M Accept: */*^M Client-ip: 10.244.1.23^M Via: http/1.1 aafb240ac1ca[0AF402FE] (ApacheTrafficServer/5.3.1)^M ^M ... [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] State Transition: SM_ACTION_DNS_LOOKUP -> SM_ACTION_CACHE_ISSUE_WRITE [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_cache_write) [119] writing to cache with URL https://urldefense.proofpoint.com/v2/url?u=http-3A__vcache-2Dedge-2D1.cdn.lab-3A31695_player.html&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=PRc1AEP4iiVziilFvhhtIHYyXu6k6Ouc9szT1KAsTrU&e= [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_cache) [119] [&HttpCacheSM::state_cache_open_write, CACHE_EVENT_OPEN_WRITE] [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] [HttpSM::main_handler, CACHE_EVENT_OPEN_WRITE] [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] [&HttpSM : state_cache_open_write, CACHE_EVENT_OPEN_WRITE] [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_trans) Next action next; __null [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] State Transition: SM_ACTION_CACHE_ISSUE_WRITE -> SM_ACTION_ORIGIN_SERVER_OPEN [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_track) entered inside do_http_server_open ][IPv4] [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] open connection to vcache-mid-1.cdn.lab: 172.15.0.170:31695 [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_seq) [HttpSM::do_http_server_open] Sending request to server [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) calling netProcessor.connect_re [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] [HttpSM::main_handler, NET_EVENT_OPEN] [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_track) entered inside state_http_server_open [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http) [119] [&HttpSM::state_http_server_open, NET_EVENT_OPEN] [Jun 25 14:19:27.053] Server {0x2b33b0df7700} DEBUG: (http_ss) [117] session born, netvc 0x2b33cc015df0 +++++++++ Proxy's Request after hooks +++++++++ -- State Machine Id: 119 GET https://urldefense.proofpoint.com/v2/url?u=http-3A__originserver.cdn.lab-3A8080_player.html&d=DwIFAg&c=jf_iaSHvJObTbx-siA1ZOg&r=7RXV34C8Z5v1pmzYtUcznQ&m=FF_sxMozrmJEyjv-p_tenc3l62dR_1TzbebFnqzkd48&s=KazpDdaeY10A52MJC1xVu-b8SW4yzRLgWvkw4x_mcg4&e= HTTP/1.1^M Host: vcache-edge-1.cdn.lab:31695^M X-Request-ID: 110fe9e2b1e8ba555c6b962d5b65b970^M X-Real-IP: 10.244.0.0^M X-Forwarded-For: 10.244.0.0, 10.244.1.23^M X-Forwarded-Host: vcache-edge-1.cdn.lab:31695^M X-Forwarded-Port: 80^M X-Forwarded-Proto: http^M X-Scheme: http^M User-Agent: curl/7.58.0^M Accept: */*^M Client-ip: 10.244.1.23^M Via: http/1.1 aafb240ac1ca[0AF402FE] (ApacheTrafficServer/5.3.1)^M ^M ... [Jun 25 14:19:27.054] Server {0x2b33ae02cac0} DEBUG: (dns) SUCCESS result for vcache-mid-1.cdn.lab = 172.15.0.170 retry 0 [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (dns) called back continuation for vcache-mid-1.cdn.lab [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http) [119] [HttpSM::main_handler, VC_EVENT_READ_READY] [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http) [119] [&HttpSM::state_read_server_response_header, VC_EVENT_READ_READY] [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http_seq) Done parsing server response header [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http_redirect) [HttpTunnel::deallocate_postdata_copy_buffers] [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http_trans) [HttpTransact::HandleResponse] [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http_seq) [HttpTransact::HandleResponse] Response received [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http_trans) [ink_cluster_time] local: 1593094767, highest_delta: 0, cluster: 1593094767 [Jun 25 14:19:27.054] Server {0x2b33b0df7700} DEBUG: (http_trans) [HandleResponse] response_received_time: 1593094767 +++++++++ Incoming O.S. Response +++++++++ -- State Machine Id: 119 HTTP/1.1 404 Not Found^M Server: nginx/1.19.0^M Date: Thu, 25 Jun 2020 14:19:27 GMT^M Content-Type: text/html^M Content-Length: 153^M Connection: keep-alive^M ^M Thanks, - Avi Weit
