[U-Boot] UBIFS Problems with U-boot 2018.1 & 4.14 Linux
Hi There ! I'm seeing a strange problem with u-boot 2018.1 and Linux 4.14 (Xilinx Petalinux 2018.1). If I write a ubifs image to flash using Linux 4.9 I can mount and read files from the image in both u-boot 2018.1 and Linux 4.14. However as soon as I write a new file to the file-system from Linux, U-boot can no longer read the file. The filesystem still mounts, but when I attempt to read the file I get the following error UBIFS error (ubi0:0 pid 0): read_block: bad data node (block 661, inode 5763) magic 0x6101831 crc0x8e6aff1a node_type 1 (data node) group_type 0 (no node group) sqnum 63819 len3075 key(5763, data, 661) size 4096 compr_typ 1 data size 3027 data: UBIFS error (ubi0:0 pid 0): do_readpage: cannot read page 661 of inode 5763, error -22 The file can still be read correctly from Linux, leading me to believe there is some form of incompatibility going on. I noticed that the ubifs version number was bumped from 4 to 5 in this commit http://git.infradead.org/linux-ubifs.git/commit/fc4b891bbefa73b496bb44b076bb5274b6bfba68 Both Linux 4.9 and U-boot 18.1 still have version 4. Could that have something do to with it ? I'm running on a ZynqMP platform with Micron Flash (On-Die ECC) Thanks in advance, /Otto ___ U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
Re: [U-Boot] UBIFS Problems with U-boot 2018.1 & 4.14 Linux
Hallo Heiko & Richard ! Turns out the len and out_len do not match, much like you suspected. Out_len is 2 bytes short (4094 vs 4096) See log below UBIFS DBG tnc: search key (5725, data, 124) UBIFS DBG tnc: found 1, lvl 0, n 2 UBIFS DBG tnc: LEB 566:61864, key (5725, data, 124) UBIFS DBG io: LEB 566:61864, data node, length 3635 UBIFS DBG tnc: search key (5725, data, 125) UBIFS DBG tnc: found 1, lvl 0, n 3 UBIFS DBG tnc: LEB 566:65504, key (5725, data, 125) UBIFS DBG io: LEB 566:65504, data node, length 3411 UBIFS DBG tnc: search key (5725, data, 126) UBIFS DBG tnc: found 1, lvl 0, n 4 UBIFS DBG tnc: LEB 566:68920, key (5725, data, 126) UBIFS DBG io: LEB 566:68920, data node, length 3041 UBIFS DBG tnc: search key (5725, data, 127) UBIFS DBG tnc: found 1, lvl 0, n 5 UBIFS DBG tnc: LEB 566:71968, key (5725, data, 127) UBIFS DBG io: LEB 566:71968, data node, length 4144 UBIFS DBG tnc: search key (5725, data, 128) UBIFS DBG tnc: found 1, lvl 0, n 6 UBIFS DBG tnc: LEB 566:76112, key (5725, data, 128) UBIFS DBG io: LEB 566:76112, data node, length 3818 UBIFS DBG tnc: search key (5725, data, 129) UBIFS DBG tnc: found 1, lvl 0, n 7 UBIFS DBG tnc: LEB 566:79936, key (5725, data, 129) UBIFS DBG io: LEB 566:79936, data node, length 4061 UBIFS DBG tnc: search key (5725, data, 130) UBIFS DBG io: LEB 560:17120, indexing node, length 188 UBIFS DBG tnc: LEB 560:17120, level 0, 8 branch UBIFS DBG tnc: found 1, lvl 0, n 0 UBIFS DBG tnc: LEB 566:84000, key (5725, data, 130) UBIFS DBG io: LEB 566:84000, data node, length 3212 UBIFS DBG tnc: search key (5725, data, 131) UBIFS DBG tnc: found 1, lvl 0, n 1 UBIFS DBG tnc: LEB 566:87216, key (5725, data, 131) UBIFS DBG io: LEB 566:87216, data node, length 3121 ubifs_decompress RC: 0 len: 4096 out_len: 4094 UBIFS error (ubi0:0 pid 0): read_block: bad data node (block 131, inode 5725) magic 0x6101831 crc0x2503e06e node_type 1 (data node) group_type 0 (no node group) sqnum 59095 len3121 key(5725, data, 131) size 4096 compr_typ 1 data size 3073 data: UBIFS error (ubi0:0 pid 0): do_readpage: cannot read page 131 of inode 5725, error -22 Error reading file '/boot/Image' If I write the file system in 4.14 and attempt to mount it on 4.9 I get root@bronte2:~# ubiattach /dev/ubi_ctrl -p /dev/mtd0 -d 0 #Create /dev/ubi8 [ 39.212876] ubi0 error: ubi_attach: PEB 51 contains corrupted VID header, and the data does not contain all 0xFF [ 39.222999] ubi0 error: ubi_attach: this may be a non-UBI PEB or a severe VID header corruption which requires manual inspection [ 39.234526] Volume identifier header dump: [ 39.238601] magic 55424921 [ 39.241725] version 1 [ 39.244232] vol_type 1 [ 39.246758] copy_flag 0 [ 39.249278] compat0 [ 39.251783] vol_id0 [ 39.254311] lnum 49 [ 39.256914] data_size 0 [ 39.259429] used_ebs 0 [ 39.261954] data_pad 0 [ 39.264464] sqnum 0 [ 39.266986] hdr_crc 9a7266c9 [ 39.270108] Volume identifier header hexdump: [ 39.274470] hexdump of PEB 51 offset 4096, length 126976[ 39.871671] ubi0 error: ubi_attach: 1 PEBs are corrupted and preserved [ 39.878132] Corrupted PEBs are: 51 [ 39.889469] UBI assert failed in ubi_wl_init at 1666 (pid 2455) Which is the same error I get if I try to mount the file system in u-boot 2018.1 Also, If I attempt to read a file that was written using 4.14 running 4.9 Linux I get the same error that 2018.1 U-boot reported, except I also get the hexdump. See below root@bronte2:~# cp /mnt/ubifs2/boot/Image /dev/null [ 871.157694] UBIFS error (ubi1:0 pid 2566): do_readpage: bad data node (block 131, inode 5725) [ 871.166166] magic 0x6101831 [ 871.169812] crc0x2503e06e [ 871.173527] node_type 1 (data node) [ 871.177527] group_type 0 (no node group) [ 871.181870] sqnum 59095 [ 871.185157] len3121 [ 871.188362] key(5725, data, 131) [ 871.192709] size 4096 [ 871.195909] compr_typ 1 [ 871.198876] data size 3073 [ 871.202080] data: [ 871.204077] : 00 b7 23 04 01 8b 21 0c 03 8b 23 4e 00 d0 1f 10 01 eb e1 fd ff 54 63 60 3a 91 e0 29 80 d2 63 68 [ 871.214756] 0020: 00 91 ed ff ff 17 1f 20 03 d5 e0 2f 8a d2 20 00 a0 f2 3f 00 00 eb cd e7 ff 54 e3 2f 8a 92 c3 ff [ 871.225434] 0040: bf f2 21 00 03 8b e7 04 00 91 3f 00 00 eb ac ff ff 54 37 ff ff 17 00 00 00 00 fd 7b bd a9 fd 03 [ 871.236111] 0060: 00 91 f3 53 01 a9 f3 03 00 aa f5 13 00 f9 f4 03 01 aa e0 03 01 aa f5 03 02 aa 61 02 00 f9 21 00 [ 871.246787] 0080: 40 f9 20 00 3f d6 60 d6 00 a9 20 00 80 d2 81 16 40 b9 00 20 c1 9a 00 04 00 d1 60 fe 01 a9 f3 53 [ 871.257461] 00a0: 41 a9 f5 13 40 f9 fd 7b c3 a8 c0 03 5f d6 1f 20 03 d5 fd 7b be a9 fd 03 00 91 f3 0b 00 f9 f3 03 [ 871.268140] 00c0: 00 aa 01 00 40 f9 e0 03 01 a
Re: [U-Boot] UBIFS Problems with U-boot 2018.1 & 4.14 Linux
Hi Richard ! Correct, the first error is printed in U-boot 2018.1 (which also matches the error I see if I try to read the file in Linux 4.9). The ubi file is created using Yocto 2.4.1 (comes with PetaLinux 2018.1) with the following settings MKUBIFS_ARGS = " -m 2048 -e 126976 -c 4096 -x lzo " UBINIZE_ARGS = " -m 2048 -p 128KiB" I will send you a separate email with a dropbox link containing the ubi file, as I would prefer not to have it posted publicly To eliminate the file generation from the equation I just created an empty file system while running Linux 4.14. This time it fails on the decompression in both u-boot & the older 4.9 Linux.See all steps below. By the way, I'm running MTD tools 2.0.0 on Linux 4.14 nandtest runs OK root@bronte2:~# nandtest /dev/mtd0 ECC corrections: 0 ECC failures : 0 Bad blocks : 0 BBT blocks : 0 0bfe: checking...of 4)... Finished pass 1 successfully -- Create Fresh UBIFS file system in Linux 4.14 - root@bronte2:~# flash_eraseall /dev/mtd0 root@bronte2:~# ubiformat -y /dev/mtd0 ubiformat: mtd0 (nand), size 201326592 bytes (192.0 MiB), 1536 eraseblocks of 131072 bytes (128.0 KiB), min. I/O size 2048 bytes libscan: scanning eraseblock 1535 -- 100 % complete ubiformat: 1536 eraseblocks have valid erase counter, mean value is 1 ubiformat: formatting eraseblock 1535 -- 100 % complete root@bronte2:~# ubiattach -p /dev/mtd0 UBI device number 0, total 1536 LEBs (195035136 bytes, 186.0 MiB), available 1452 LEBs (184369152 bytes, 175.8 MiB), LEB size 126976 bytes (124.0 KiB) [ 327.214676] ubi0: attaching mtd0 [ 327.512205] ubi0: scanning is finished [ 327.520635] ubi0: attached mtd0 (name "rootfs1", size 192 MiB) [ 327.520640] ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes [ 327.520643] ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 [ 327.520647] ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096 [ 327.520650] ubi0: good PEBs: 1536, bad PEBs: 0, corrupted PEBs: 0 [ 327.520653] ubi0: user volume: 0, internal volumes: 1, max. volumes count: 128 [ 327.520656] ubi0: max/mean erase counter: 3/2, WL threshold: 4096, image sequence number: 679562370 [ 327.520660] ubi0: available PEBs: 1452, total reserved PEBs: 84, PEBs reserved for bad PEB handling: 80 [ 327.520671] ubi0: background thread "ubi_bgt0d" started, PID 2723 root@bronte2:~# ubimkvol /dev/ubi0 -N rootfs -m Set volume size to 184369152 Volume ID 0, size 1452 LEBs (184369152 bytes, 175.8 MiB), LEB size 126976 bytes (124.0 KiB), dynamic, name "rootfs", alignment 1 [ 388.953037] UBIFS (ubi0:0): default file-system created [ 388.953332] UBIFS (ubi0:0): background thread "ubifs_bgt0_0" started, PID 2737 [ 388.986507] UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "rootfs" [ 388.986514] UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes [ 388.986519] UBIFS (ubi0:0): FS size: 182845440 bytes (174 MiB, 1440 LEBs), journal size 9142272 bytes (8 MiB, 72 LEBs) [ 388.986522] UBIFS (ubi0:0): reserved for root: 4952683 bytes (4836 KiB) [ 388.986528] UBIFS (ubi0:0): media format: w5/r0 (latest is w5/r0), UUID 9F3EA589-6F91-42DF-BC93-216FB3868A2D, small LPT model root@bronte2:~# mkdir /mnt/ubifs1/boot root@bronte2:~# mount -t ubifs /dev/ubi0_0 /mnt/ubifs1 root@bronte2:~# cp /boot/* /mnt/ubifs1/boot/ root@bronte2:~# sync [Reboot into u-boot 2018.1] --- Attempting to read files back in U-boot 2018.1 -- ZynqMP> ubi part rootfs1 Bad block table found at page 262080, version 0x01 Bad block table found at page 262016, version 0x01 ubi0: attaching mtd2 ubi0: scanning is finished ubi0: attached mtd2 (name "mtd=0", size 192 MiB) ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096 ubi0: good PEBs: 1536, bad PEBs: 0, corrupted PEBs: 0 ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128 ubi0: max/mean erase counter: 4/2, WL threshold: 4096, image sequence number: 679562370 ubi0: available PEBs: 80, total reserved PEBs: 1456, PEBs reserved for bad PEB handling: 0 ZynqMP> ubifsmount ubi0:rootfs [Cut out lots of debug msgs] UBIFS (ubi0:0): UBIFS: mounted UBI device 0, volume 0, name "rootfs", R/O mode UBIFS (ubi0:0): LEB size: 126976 bytes (124 KiB), min./max. I/O unit sizes: 2048 bytes/2048 bytes UBIFS (ubi0:0): FS size: 182845440 bytes (174 MiB, 1440 LEBs), journal size 9142272 bytes (8 MiB, 72 LEBs) UBIFS (ubi0:0): reserved for root: 4952683 bytes (4836 KiB) UBIFS (ubi0:0): media format: w5/r0 (latest is w4/r0), UUID 9f3ea589-6f91-42df-bc93-216fb3868a2d, small LPT model ZynqMP> ubifsload 0x1000 /boot/Image Loading file '/boot/Image' to addr 0x1000... UBIFS DBG tnc: (lowest) (1, direntry, 0x00) UBIFS DBG tnc: search key (1, direntry, 0x00) UBIFS DBG tnc: found 0, lvl 0, n 0 UBIFS DBG
Re: [U-Boot] UBIFS Problems with U-boot 2018.1 & 4.14 Linux
Hi Richard ! To summarize the observations from the last few days. * Linux 4.9 & U-boot 2018 behave the same when attempting to read from a ubifs file system * Whenever Linux 4.14 writes to a ubifs there is some form of incompatibility introduced causing assorted errors when the file system is read in Linux 4.9 or U-boot. However the files can still be read in Linux 4.14. This lead me to suspect a problem with the OOB data. So I copied over a 16MB textfile to a freshly minted no-compression ubifs partition (running 4.14) and ran a nanddump both from Linux 4.9 and 4.14 and the contents are different - Bit errors are present in the 4.9-dump.. See https://www.dropbox.com/s/sbi64wfqi62noxi/bit_error.png?dl=0 So I tried to write a 94Mb file directly to the NAND using flash_eraseall /dev/mtd2 nandwrite -p /dev/mtd2 Bronte2_fw_20180509-r.bin nanddump -f mtd2.dump /dev/mtd2 -l 94103952 And to my big surprise the file written in 4.14 reads back identically in 4.9 and vice versa. Any idea on what could be causing something like this ? Does ubifs use a different writing method compared to nandwrite ? Its almost like the two have different ECC / OOB processing Thanks for your time ! /Otto On Sat, May 19, 2018 at 1:37 AM, Richard Weinberger wrote: > Am Samstag, 19. Mai 2018, 01:56:33 CEST schrieb Otto Blom: >> UBIFS error (ubi0:0 pid 0): crypto_comp_decompress: cannot decompress >> 2801 bytes, compressor lzo, error -6 > > LZO_E_LOOKBEHIND_OVERRUN... > >> UBIFS error (ubi0:0 pid 0): ubifs_decompress: cannot decompress 2801 >> bytes, compressor lzo, error -6 >> ubifs_decompress RC: -6 len: 4096 out_len: 3377 >> UBIFS error (ubi0:0 pid 0): read_block: bad data node (block 3, inode 68) >> magic 0x6101831 >> crc0xf15144c6 >> node_type 1 (data node) >> group_type 0 (no node group) >> sqnum 3276 >> len2849 >> key(68, data, 3) >> size 4096 >> compr_typ 1 >> data size 2801 >> data: >> UBIFS error (ubi0:0 pid 0): do_readpage: cannot read page 3 of inode >> 68, error -22 >> Error reading file '/boot/Image' > > So, LZO *does* return an error. > Did you edit the logs in your last mails? Or was some U-Boot silencing option > enabled? I did not edit the logs. Under the previous scenario ubifs_decompress returned zero, but there was a length mismatch (len != len_out). The difference between the two scenarios is that now I'm using a freshly formatted partition that was initialized while running Linux 4.14. Previously I was testing with the a rootfs being written using the -f option of ubiformat, followed by updating a single file (/boot/Image). Purpose of the later test was to eliminate the ubi-file being part of the problem. > Does it work if you disable compression? I tried disabling compression and also switched to using a large (16Mb) text file so it would be easier to see any corruption. Still fails unfortunately.. See below for the full log. The thing that I find the most strange is that the file can still be read correctly in Linux 4.14.. Not sure if the nanddump of the ubifs partition is of any value, but I zipped it up and put it on dropbox just in case https://www.dropbox.com/s/sd8ctph9ie7u20m/rootfs.zip?dl=0 root@bronte2:~# flash_eraseall /dev/mtd2 flash_eraseall has been replaced by `flash_erase 0 0`; please use it Erasing 128 Kibyte @ 7f6 -- 99 % complete flash_erase: Skipping bad block at 07f8 flash_erase: Skipping bad block at 07fa flash_erase: Skipping bad block at 07fc flash_erase: Skipping bad block at 07fe Erasing 128 Kibyte @ 7fe -- 100 % complete ubiattach /dev/ubi_ctrl -p /dev/mtd2 -d 2 ubimkvol /dev/ubi2 -N rootfs -m mkdir -p /mnt/ubifs3 mount -t ubifs -o compr=none /dev/ubi2_0 /mnt/ubifs3 cp /tmp/iliad_big.txt /mnt/ubifs3/iliad_16M.txt umount /mnt/ubifs3/ ubidetach -p /dev/mtd2 nanddump -f rootfs.ubi /dev/mtd2# < File on dropbox [Booting into u-boot 2018.1] ZynqMP> ubi part misc Bad block table found at page 262080, version 0x01 Bad block table found at page 262016, version 0x01 ubi0: attaching mtd2 ubi0: scanning is finished ubi0 warning: ubi_calculate_reserved: number of bad PEBs (4) is above the expected limit (0), not reserving any PEBs for bad PEB handling, will use available PEBs (if any) ubi0: attached mtd2 (name "mtd=2", size 128 MiB) ubi0: PEB size: 131072 bytes (128 KiB), LEB size: 126976 bytes ubi0: min./max. I/O unit sizes: 2048/2048, sub-page size 2048 ubi0: VID header offset: 2048 (aligned 2048), data offset: 4096 ubi0: good PEBs: 1020, bad PEBs: 4, corrupted PEBs: 0 ubi0: user volume: 1, internal volumes: 1, max. volumes count: 128 ubi0: max/mean erase counter: 2/1, WL threshold:
Re: [U-Boot] UBIFS Problems with U-boot 2018.1 & 4.14 Linux
Thanks a lot for your help Richard & Heiko ! I appreciate it I agree, this does not look u-boot or ubifs related anymore /Otto On Tue, May 22, 2018 at 2:31 AM, Heiko Schocher wrote: > Hello Otto, > > Am 22.05.2018 um 08:24 schrieb Richard Weinberger: >> >> Otto, >> >> Am Dienstag, 22. Mai 2018, 03:30:04 CEST schrieb Otto Blom: >>> >>> Hi Richard ! >>> >>> To summarize the observations from the last few days. >>> >>> * Linux 4.9 & U-boot 2018 behave the same when attempting to read from >>> a ubifs file system >>> * Whenever Linux 4.14 writes to a ubifs there is some form of >>> incompatibility introduced >>> causing assorted errors when the file system is read in Linux 4.9 or >>> U-boot. However >>> the files can still be read in Linux 4.14. >>> >>> This lead me to suspect a problem with the OOB data. So I copied over >>> a 16MB textfile to >>> a freshly minted no-compression ubifs partition (running 4.14) and ran >>> a nanddump both from >>> Linux 4.9 and 4.14 and the contents are different - Bit errors are >>> present in the 4.9-dump.. >> >> >> This smells like a problem on MTD/NAND-Level. >> Decompression might fail because bits changes but didn't get fixed by your >> driver. > > > +1 from my side ... > > > bye, > Heiko > -- > DENX Software Engineering GmbH, Managing Director: Wolfgang Denk > HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany > Phone: +49-8142-66989-52 Fax: +49-8142-66989-80 Email: h...@denx.de ___ U-Boot mailing list U-Boot@lists.denx.de https://lists.denx.de/listinfo/u-boot
