Andreas Enge <andr...@enge.fr> writes: > Am Tue, Oct 29, 2024 at 12:42:32PM +0100 schrieb Andreas Enge: >> I have restarted nginx and nar-herder (just an educated guess), but >> to no avail. > > I think the problem is further upstream, in the place where the nars are > actually stored (hatysa). When I download a file that is in the cache, > such as > wget > https://bordeaux.guix.gnu.org/nar/zstd/000z8mclk0p7jmrijj6l54bpnam29bqj-r-minfi-1.48.0 > it works.
Hmm, yeah the nar-herder/nginx on hatysa seems to be stuck, it looks like quite a low level problem since I see btrfs related errors in /var/log/messages Oct 29 10:14:07 localhost vmunix: [2910798.796657] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: Oct 29 10:14:07 localhost vmunix: [2910798.802941] rcu: Tasks blocked on level-0 rcu_node (CPUs 0-15): P30086 Oct 29 10:14:07 localhost vmunix: [2910798.809730] rcu: (detected by 11, t=5252 jiffies, g=253094433, q=9317 ncpus=16) Oct 29 10:14:07 localhost vmunix: [2910798.817292] task:nginx state:R running task stack:0 pid:30086 tgid:30086 ppid:30085 flags:0x0000000c Oct 29 10:14:07 localhost vmunix: [2910798.828334] Call trace: Oct 29 10:14:07 localhost vmunix: [2910798.830945] dump_backtrace+0x90/0xe8 Oct 29 10:14:07 localhost vmunix: [2910798.834789] show_stack+0x18/0x24 Oct 29 10:14:07 localhost vmunix: [2910798.838273] sched_show_task+0x12c/0x18c Oct 29 10:14:07 localhost vmunix: [2910798.842364] rcu_sched_clock_irq+0xd00/0xdcc Oct 29 10:14:07 localhost vmunix: [2910798.846804] update_process_times+0x68/0xac Oct 29 10:14:07 localhost vmunix: [2910798.851155] tick_sched_handle+0x34/0x58 Oct 29 10:14:07 localhost vmunix: [2910798.855245] tick_nohz_highres_handler+0x50/0xa8 Oct 29 10:14:07 localhost vmunix: [2910798.860030] __hrtimer_run_queues+0x138/0x1b0 Oct 29 10:14:07 localhost vmunix: [2910798.864556] hrtimer_interrupt+0xe8/0x244 Oct 29 10:14:07 localhost vmunix: [2910798.868736] arch_timer_handler_phys+0x34/0x44 Oct 29 10:14:07 localhost vmunix: [2910798.873349] handle_percpu_devid_irq+0x84/0x130 Oct 29 10:14:07 localhost vmunix: [2910798.878048] generic_handle_domain_irq+0x2c/0x44 Oct 29 10:14:07 localhost vmunix: [2910798.882834] gic_handle_irq+0x4c/0x110 Oct 29 10:14:07 localhost vmunix: [2910798.886748] call_on_irq_stack+0x24/0x4c Oct 29 10:14:07 localhost vmunix: [2910798.890839] do_interrupt_handler+0x80/0x84 Oct 29 10:14:07 localhost vmunix: [2910798.895189] el1_interrupt+0x34/0x68 Oct 29 10:14:07 localhost vmunix: [2910798.898934] el1h_64_irq_handler+0x18/0x24 Oct 29 10:14:07 localhost vmunix: [2910798.903195] el1h_64_irq+0x64/0x68 Oct 29 10:14:07 localhost vmunix: [2910798.906763] xas_descend+0x20/0x8c Oct 29 10:14:07 localhost vmunix: [2910798.910332] filemap_get_entry+0x5c/0x1ac Oct 29 10:14:07 localhost vmunix: [2910798.914513] __filemap_get_folio+0x4c/0x394 Oct 29 10:14:07 localhost vmunix: [2910798.918864] alloc_extent_buffer+0x210/0x860 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.924037] read_tree_block+0x20/0x16c [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.928766] btrfs_release_path+0x2ac/0x3b4 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.933843] btrfs_search_slot+0x340/0xb24 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.938835] btrfs_delete_one_dir_name+0x194/0x4c8 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.944516] btrfs_lookup_bio_sums+0x1e4/0x40c [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.949854] __btrfs_panic+0xb1c/0xec4 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.954498] btrfs_submit_bio+0x28/0x40 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.959224] ordered_data_exit+0x638/0x2e28 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.964300] extent_readahead+0x34c/0x384 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.969203] btrfs_transaction_exit+0x1a3c/0x3f08 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910798.974796] read_pages+0x80/0x2e0 Oct 29 10:14:07 localhost vmunix: [2910798.978374] page_cache_ra_unbounded+0x160/0x1e4 Oct 29 10:14:07 localhost vmunix: [2910798.983159] page_cache_ra_order+0x8c/0x2f8 Oct 29 10:14:07 localhost vmunix: [2910798.987510] ondemand_readahead+0x170/0x30c Oct 29 10:14:07 localhost vmunix: [2910798.991859] page_cache_sync_ra+0x74/0xa4 Oct 29 10:14:07 localhost vmunix: [2910798.996036] filemap_get_pages+0xc4/0x66c Oct 29 10:14:07 localhost vmunix: [2910799.000215] filemap_read+0xbc/0x398 Oct 29 10:14:07 localhost vmunix: [2910799.003959] btrfs_release_file+0xc84/0x13d8 [btrfs] Oct 29 10:14:07 localhost vmunix: [2910799.009128] vfs_read+0x24c/0x2ec Oct 29 10:14:07 localhost vmunix: [2910799.012617] ksys_pread64+0x78/0xbc Oct 29 10:14:07 localhost vmunix: [2910799.016271] __arm64_sys_pread64+0x20/0x2c Oct 29 10:14:07 localhost vmunix: [2910799.020533] invoke_syscall+0x48/0x118 Oct 29 10:14:07 localhost vmunix: [2910799.024451] el0_svc_common.constprop.0+0x40/0xe8 Oct 29 10:14:07 localhost vmunix: [2910799.029322] do_el0_svc+0x20/0x2c Oct 29 10:14:07 localhost vmunix: [2910799.032803] el0_svc+0x34/0xb8 Oct 29 10:14:07 localhost vmunix: [2910799.036025] el0t_64_sync_handler+0x13c/0x158 Oct 29 10:14:07 localhost vmunix: [2910799.040547] el0t_64_sync+0x190/0x194 > As far as I know, only Chris has access to this machine. We should be able to get other people SSH access, I'll try restarting the machine.
signature.asc
Description: PGP signature
