I've run into a very annoying problem with my swift cluster and I'm hoping somebody can help me out. During any given 100 uploads, I'll usually see between 1 to 4% of the calls give a 404. The swift client writes the following:
Object PUT failed: http://swift:8080/v1/AUTH_swift/test1/042 404 Not Found [first 60 chars of response] <html><h1>Not Found</h1><p>The resource could not be found.< I'm uploading everything to a single bucket and there's only 100 files in there that are less than 10kb each. For auth I'm using tempauth. The 404 is totally random. Sometimes I don't see it, sometimes I see it 5% of the time. On the proxies, the error message looks like this: Dec 23 09:00:08 proxy01 proxy-server 10.xx.xx.xx 10.xxx.xxx.xxx 23/Dec/2013/09/00/08 PUT /v1/AUTH_swift/test1/002 HTTP/1.0 404 - - AUTH_tk275e94194c2447a787e57ec2574789f3 - 70 - txf2066f1fa5ec403db9791-0052b7fb98 - 0.0367 - - On the storage servers I see this: Dec 23 08:49:11 storage1 container-server 10.xx.xx.xxx - - [23/Dec/2013:08:49:11 +0000] "HEAD /slot-8/11126/AUTH_swift/test1" 404 - "tx92750eeef61640a3929f4-0052b7f907" "HEAD http://swift:8080/v1/AUTH_swift/test1"; "proxy-server 10178" 0.0002 Note that these are not necessarily from the same request, but it's the messages that pop up. Here's what I have for my proxy-server.conf: [DEFAULT] bind_port = 8080 workers = 8 user = swift log_statsd_host = statsd log_statsd_port = 8125 log_statsd_default_sample_rate = 1 log_statsd_metric_prefix = proxy01 [pipeline:main] pipeline = healthcheck proxy-logging cache swift3 tempauth proxy-logging proxy-server [app:proxy-server] use = egg:swift#proxy allow_account_management = true account_autocreate = true [filter:proxy-logging] use = egg:swift#proxy_logging [filter:swift3] use = egg:swift3#swift3 [filter:tempauth] use = egg:swift#tempauth [some user] .reseller_admin [filter:healthcheck] use = egg:swift#healthcheck [filter:cache] use = egg:swift#memcache memcache_servers = 10.xx.xx.xx:11211,10.xx.xx.xx:11211,10.xx.xx.xx:11211, 10.xx.xx.xx:11211,10.xx.xx.xx:11211 On the storage backends I'm using the following: obect-server.conf [DEFAULT] bind_ip = 10.xx.xx.xxx workers = 16 log_facility = LOG_LOCAL4 pipeline:main] pipeline = object-server app:object-server] use = egg:swift#object [object-replicator] [object-updater] [object-auditor] The container and account servers look identical to this. Any ideas? -- Stephen Wood www.heystephenwood.com
_______________________________________________ Mailing list: http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack Post to : openstack@lists.openstack.org Unsubscribe : http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack
