ZFS out of swap space
-- Hello. Please help . I mirror ZFS 9.3 , after an active it by using mysql read \ write from an external script something broken. The operating system is not loaded at the time of "Mount local filesystems" pool consists of a mirror (raid 1 ) + hot swap, zfs partitions on a separate . zpool import -f -R /tmp zroot freezes try to import a pool from a LiveCD (10.1) -> zpool import -f -R / tmp zroot out as in deep thought and then wrote something like pid 99217 (sh), uid 0, was killed: out of swap space pid 896 (ssh), uid 0, was killed: out of swap space zpool made of 2 disks to GPT and one as hot-swap. zpool status -v pool: zroot state: ONLINE scan: none requested config: NAME STATE READ WRITE CKSUM zroot ONLINE 0 0 0 mirror-ONLINE 0 0 0 0 gpt / disk0 ONLINE 0 0 0 gpt / disk1 ONLINE 0 0 0 spares gpt / disk2 AVAIL errors: No known data errors On the server 4096 ram, zfs datasets are divided like this: zroot 57.8G 212G 170M / zroot / tmp 18.2M 212G 18.2M / tmp zroot / usr 15.4G 212G 409M / usr zroot / usr / home 6.54G 212G 6.54G / usr / home zroot / usr / local 3.78G 212G 3.78G / usr / local zroot / usr / obj 432K 212G 432K / usr / obj zroot / usr / ports 2.61G 212G 1.69G / usr / ports zroot / usr / ports / distfiles 815M 212G 815M / usr / ports / distfiles zroot / usr / ports / packages 126M 212G 126M / usr / ports / packages zroot / usr / src 2.08G 212G 2.08G / usr / src zroot / var 41.9G 212G 194M / var zroot / var / backups 147M 212G 147M / var / backups zroot / var / crash 224K 212G 224K / var / crash zroot / var / db 41.6G 212G 331M / var / db zroot / var / db / mysql 41.2G 212G 53.6M / var / db / mysql zroot / var / db / mysql / billing 37.8G 212G 37.8G / var / db / mysql / billing zroot / var / db / mysql / ibdata 3.05G 212G 3.05G / var / db / mysql / ibdata zroot / var / db / mysql / logs 230M 212G 230M / var / db / mysql / logs zroot / var / db / pkg 93.0M 212G 93.0M / var / db / pkg zroot / var / empty 216K 212G 216K / var / empty zroot / var / log 39.2M 212G 39.2M / var / log zroot / var / mail 216K 212G 216K / var / mail zroot / var / run 452K 212G 452K / var / run zroot / var / tmp 476K 212G 476K / var / tmp I tried to mount separately mount -t zfs zroot/var/db/mysql/billing Please see screenshot http://s17.postimg.org/am4nuw9r3/dsg.jpg Now I did zpool import -f zroot and zpool scrub zroot, Is already 11 percent of 11 gigabytes of 106 , but I do not think it will help . Scrub shows have 327 % 348 gigabytes from 106 gigabytes and scan is slow, no estimated time. Please help colleagues ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
Now the system is booted from the LiveCD (10.1) and I imported the pool using zpool import -f zroot, but if you try to mount everything freezes . What will swap on the LiveCD if I did not do it there ? psstat -T 38/130030 files 0M / 0M swap space swapinfo - now empty. Scrub shows 540 % complete , what the hell does that mean? Of course I was standing on a running system explanatory variables vfs.zfs.arc_max = 2147483648 Now LiveCD is set vfs.zfs.arc_max = 3050086400 The question is, how do I recover the operating system now? Среда, 25 марта 2015, 11:35 -04:00 от Robert Blayzor : >On Mar 25, 2015, at 11:08 AM, armonia < armo...@inbox.ru > wrote: >> -- Hello. Please help . I mirror ZFS 9.3 , after an active it by using mysql >> read \ write from an external script something broken. The operating system >> is not loaded at the time of "Mount local filesystems" >> >> pool consists of a mirror (raid 1 ) + hot swap, zfs partitions on a separate >> . >> >> zpool import -f -R /tmp zroot freezes > > >Where is your swap partition? > >My guess is you are hitting swap because you need to tune vfs.zfs.arc_max to >allow room for the OS and any running applications. If you do not set arc_max >then ZFS will basically consume all RAM save for about 1GB for the OS and any >apps. > >Maybe something like the following to /boot/loader.conf may help... (or not) > >vfs.zfs.arc_max="2G" > > >-- >Robert >inoc.net!rblayzor >http://inoc.net/ > > > -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
Thanks, I 'll try and let you know the result . Среда, 25 марта 2015, 16:49 +01:00 от Michelle Sullivan : >armonia wrote: >> >> -- Hello. Please help . I mirror ZFS 9.3 , after an active it by using mysql >> read \ write from an external script something broken. The operating system >> is not loaded at the time of "Mount local filesystems" >> >> pool consists of a mirror (raid 1 ) + hot swap, zfs partitions on a separate >> . >> >> zpool import -f -R /tmp zroot freezes >> >> try to import a pool from a LiveCD (10.1) -> zpool import -f -R / tmp zroot >> out as in deep thought and then wrote something like >> >> pid 99217 (sh), uid 0, was killed: out of swap space >> pid 896 (ssh), uid 0, was killed: out of swap space >> >> zpool made of 2 disks to GPT and one as hot-swap. >> > >Write a 11-STABLE USB drive boot from it to single user, run your >import, then run an export, then reboot into 9.3 and import without flags. > >Regards, > >-- >Michelle Sullivan >http://www.mhix.org/ > >___ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to " freebsd-stable-unsubscr...@freebsd.org " -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
When you try to import pid 32 (zpool), uid 0, was killed: out of swap space Maybe help to create swap partition directly into the LiveUSB ? Среда, 25 марта 2015, 16:49 +01:00 от Michelle Sullivan : >armonia wrote: >> >> -- Hello. Please help . I mirror ZFS 9.3 , after an active it by using mysql >> read \ write from an external script something broken. The operating system >> is not loaded at the time of "Mount local filesystems" >> >> pool consists of a mirror (raid 1 ) + hot swap, zfs partitions on a separate >> . >> >> zpool import -f -R /tmp zroot freezes >> >> try to import a pool from a LiveCD (10.1) -> zpool import -f -R / tmp zroot >> out as in deep thought and then wrote something like >> >> pid 99217 (sh), uid 0, was killed: out of swap space >> pid 896 (ssh), uid 0, was killed: out of swap space >> >> zpool made of 2 disks to GPT and one as hot-swap. >> > >Write a 11-STABLE USB drive boot from it to single user, run your >import, then run an export, then reboot into 9.3 and import without flags. > >Regards, > >-- >Michelle Sullivan >http://www.mhix.org/ > >___ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to " freebsd-stable-unsubscr...@freebsd.org " -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
Not solve the problem , I booted from 11.0 STABLE and when you try to import a pool system goes into freeze . (zpool import -f zroot) Just hang when attempting to mount the problematic section zroot / var / db / mysql / billing In this home are installed properly , I even removed through truncate -s 0 15 gigs in the file / home / user, but it does not help . More options ? All attempts to freeze mount lead as well as attempts to boot the system. Среда, 25 марта 2015, 16:49 +01:00 от Michelle Sullivan : >armonia wrote: >> >> -- Hello. Please help . I mirror ZFS 9.3 , after an active it by using mysql >> read \ write from an external script something broken. The operating system >> is not loaded at the time of "Mount local filesystems" >> >> pool consists of a mirror (raid 1 ) + hot swap, zfs partitions on a separate >> . >> >> zpool import -f -R /tmp zroot freezes >> >> try to import a pool from a LiveCD (10.1) -> zpool import -f -R / tmp zroot >> out as in deep thought and then wrote something like >> >> pid 99217 (sh), uid 0, was killed: out of swap space >> pid 896 (ssh), uid 0, was killed: out of swap space >> >> zpool made of 2 disks to GPT and one as hot-swap. >> > >Write a 11-STABLE USB drive boot from it to single user, run your >import, then run an export, then reboot into 9.3 and import without flags. > >Regards, > >-- >Michelle Sullivan >http://www.mhix.org/ > >___ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to " freebsd-stable-unsubscr...@freebsd.org " -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
After importing I press ctrl + t and here's the conclusion: load: 0.20 cmd: zpool 725 [tx->tx_sync_done_cv] 32.50r 0.00y 5.59s 0% 6432k Среда, 25 марта 2015, 16:49 +01:00 от Michelle Sullivan : >armonia wrote: >> >> -- Hello. Please help . I mirror ZFS 9.3 , after an active it by using mysql >> read \ write from an external script something broken. The operating system >> is not loaded at the time of "Mount local filesystems" >> >> pool consists of a mirror (raid 1 ) + hot swap, zfs partitions on a separate >> . >> >> zpool import -f -R /tmp zroot freezes >> >> try to import a pool from a LiveCD (10.1) -> zpool import -f -R / tmp zroot >> out as in deep thought and then wrote something like >> >> pid 99217 (sh), uid 0, was killed: out of swap space >> pid 896 (ssh), uid 0, was killed: out of swap space >> >> zpool made of 2 disks to GPT and one as hot-swap. >> > >Write a 11-STABLE USB drive boot from it to single user, run your >import, then run an export, then reboot into 9.3 and import without flags. > >Regards, > >-- >Michelle Sullivan >http://www.mhix.org/ > >___ >freebsd-stable@freebsd.org mailing list >http://lists.freebsd.org/mailman/listinfo/freebsd-stable >To unsubscribe, send any mail to " freebsd-stable-unsubscr...@freebsd.org " -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
How much are you waiting time after import ? Look a screenshot : http://i58.tinypic.com/mvkj00.jpg It helped you because you certainly do import 11 branch ? Пятница, 27 марта 2015, 13:59 +01:00 от Michelle Sullivan : >armonia wrote: >> Not solve the problem, I booted from 11.0 STABLE and when you try to >> import a pool system goes into freeze. (zpool import -f zroot) >> >> Just hang when attempting to mount the problematic section zroot / var >> / db / mysql / billing >> >> In this home are installed properly, I even removed through truncate >> -s 0 15 gigs in the file / home / user, but it does not help. >> >> More options? All attempts to freeze mount lead as well as attempts to >> boot the system. >> >All I can suggest is boot and wait - see if it comes back - if not I am >out of suggestions - "zfs import -Ff storage" worked for me when I was >running out of swap when trying to import on a 9.2 system (storage being >my pool name) > >After the import I exported booted back to 9.2 and it all worked... >issued a 'scrub' *after* booting back to 9.2 (don't do it before) > >Michelle > >> >> Среда, 25 марта 2015, 16:49 +01:00 от Michelle Sullivan >> < miche...@sorbs.net >: >> >> armonia wrote: >> > >> > -- Hello. Please help . I mirror ZFS 9.3 , after an active it by >> using mysql read \ write from an external script something broken. >> The operating system is not loaded at the time of "Mount local >> filesystems" >> > >> > pool consists of a mirror (raid 1 ) + hot swap, zfs partitions >> on a separate . >> > >> > zpool import -f -R /tmp zroot freezes >> > >> > try to import a pool from a LiveCD (10.1) -> zpool import -f -R >> / tmp zroot out as in deep thought and then wrote something like >> > >> > pid 99217 (sh), uid 0, was killed: out of swap space >> > pid 896 (ssh), uid 0, was killed: out of swap space >> > >> > zpool made of 2 disks to GPT and one as hot-swap. >> > >> >> Write a 11-STABLE USB drive boot from it to single user, run your >> import, then run an export, then reboot into 9.3 and import >> without flags. >> >> Regards, >> >> -- >> Michelle Sullivan >> http://www.mhix.org/ >> >> ___ >> freebsd-stable@freebsd.org >> < /compose?To=freebsd%2dsta...@freebsd.org > mailing list >> http://lists.freebsd.org/mailman/listinfo/freebsd-stable >> To unsubscribe, send any mail to >> " freebsd-stable-unsubscr...@freebsd.org >> < /compose?To=freebsd%2dstable%2dunsubscr...@freebsd.org >" >> >> >> >> -- > > >-- >Michelle Sullivan >http://www.mhix.org/ > -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
Michelle thank you so much , like happened. At 10.1 LiveUSB I did zpool import but probably the tenth attempt, he has executed , the same thing happened on the 11 branch. So I removed just in case bad dataset - / var / db / mysql / billing, After I did the export and rebooted in the hard disk. I am grateful to you about . Пятница, 27 марта 2015, 18:07 +01:00 от Michelle Sullivan : >armonia wrote: >> How much are you waiting time after import? > >I was waiting for 4 days for the import to complete - however I do have >a 16 drive 48T pool. > >> >> Look a screenshot: >> >> http://i58.tinypic.com/mvkj00.jpg >> >> It helped you because you certainly do import 11 branch? >> >It helped because there was some patches in the 11 branch that handle >errors in the pool better than in 9.x... when it imported I was then >able to export and restart back to 9.x which corrected the uncaught >errors and it was importable straight back into 9.x.. I was then able to >correct the rest of the errors by doing a scrub in 9.x (which ultimately >was all caused by power, ups and battery failure whilst in the middle of >a resilver. > >Regards, > >-- >Michelle Sullivan >http://www.mhix.org/ > -- ___ freebsd-stable@freebsd.org mailing list http://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
Yes, my mistake was probably that I have included data deduplication to see
how it works, but not turned it off at the right time . In this case, the
machine memory of 4 GB
async_destroy - too enabled.
That is the conclusion I have deduplication disabled .
How to import a pool of read-only ?
Thank you for your response .
zpool get all zroot
NAME PROPERTY VALUE SOURCE
zroot size 230G -
zroot capacity 24% -
zroot altroot - default
zroot health ONLINE -
zroot guid 1229884058434432944 default
zroot version - default
zroot bootfs zroot local
zroot delegation on default
zroot autoreplace on local
zroot cachefile - default
zroot failmode wait default
zroot listsnapshots on local
zroot autoexpand off default
zroot dedupditto 0 default
zroot dedupratio 1.02x -
zroot free 174G -
zroot allocated 56.1G -
zroot readonly off -
zroot comment ZFS local
zroot expandsize 0 -
zroot freeing 0 default
zroot feature@async_destroy enabled local
zroot feature@empty_bpobj active local
zroot feature@lz4_compress active local
zroot feature@multi_vdev_crash_dump enabled local
zroot feature@spacemap_histogram active local
zroot feature@enabled_txg active local
zroot feature@hole_birth active local
zroot feature@extensible_dataset enabled local
zroot feature@bookmarks enabled local
zroot feature@filesystem_limits enabled local
Понедельник, 30 марта 2015, 0:36 -07:00 от Xin Li :
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA512
>
>
>
>On 3/27/15 05:26, armonia wrote:
>> After importing I press ctrl + t and here's the conclusion:
>>
>> load: 0.20 cmd: zpool 725 [tx->tx_sync_done_cv] 32.50r 0.00y 5.59s
>> 0% 6432k
>
>Have you ever enabled e.g. dedup on certain dataset(s) and have a lot
>of files, and the pool don't have 'async destroy' feature enabled? In
>that case the fastest way to recover, if this took too long, would
>probably import the pool read-only and copy data to another pool.
>
>On -CURRENT you can use dtrace -qn 'zfs-dbgmsg{printf("%s\n",
>stringof(arg0))}' to see more verbose information, they are not always
>helpful but will give you better idea on what is going on under the hood
>.
>
>Cheers,
>-BEGIN PGP SIGNATURE-
>
>iQIcBAEBCgAGBQJVGP0IAAoJEJW2GBstM+nsjRUP/11uzAQjAhQdOKTDMYt9gbTt
>DsMn1C5x3X+btMuMyJ6JfxiOhm1qxolDEsjilodpj+i7m09a42N1GZrqEKBIY4PP
>wvVwHHGuG1+zvpcQYfyWz50lKOgSIeLY3CUkyRRLHu3XsGrhj6uVEBjJRqLzRFof
>oCTlAPeXFkZgs5wkufwJ9kx5JJT4UvZKyBbj3CNL7xMGmEIKffIMZWQv3SzIROdU
>3QLjKSxTX969l6bNPEG/Fr262SZvXq9wPF2hbXs+AgLeDduz+ILhOMF1Za9+PTu+
>U5zJM5MUEOimacAys0ldQ5kVarufFySCv1VXvmIyUPPsVu+WuBAWE7lC1c8tdNnh
>vawMSfA2z5GtowxcpRVkl0CuVdO9AHZ2cHQzqrVh+rC/3HEdpCiUYakvoU3g1LhT
>9Yo1s6dDUQcl6cfObDw5QU1eNc5dnCwENef5UnraH/GYbV4u8j4ClBPD7jw7zdlo
>jgGVMcXORJEztYW3LCQNUU0phP0c3jNm9x84Do4AhLPMTPfxPrXiHJy/BtSrDw4B
>BZlvqDRhaaOlTU/RPoAIdzFm1PztGLG2r5zrPR/ZKB3UquDctWSNz84jvgN3OK0d
>XyTsJsAH3lRLO5RNIt/Km4z86vnmL0WZdiY919XsBWlVbh1db+qKHs/KeSpaJ1Fd
>wqeqhWwU8l5yGKik19du
>=Bm6L
>-END PGP SIGNATURE-
>___
>freebsd-stable@freebsd.org mailing list
>http://lists.freebsd.org/mailman/listinfo/freebsd-stable
>To unsubscribe, send any mail to " freebsd-stable-unsubscr...@freebsd.org "
--
___
freebsd-stable@freebsd.org mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-stable
To unsubscribe, send any mail to "freebsd-stable-unsubscr...@freebsd.org"
Re[2]: ZFS out of swap space
After specifying Michelle Sullivan restore everything , now the system is
loaded .
At 10.1 LiveUSB I did zpool import but probably the tenth attempt, he has
executed , the same thing happened on the 11 branch. So I removed just in case
bad dataset - / var / db / mysql / billing,
After I did the export and rebooted in the hard disk. I am grateful to you
about .
>Can you try if setting the sysctl variable vfs.zfs.free_max_blocks to
>a limited number, like, 100,000, would make the pool import properly?
>You may have run out of memory because of too much data being free'ed
>in one transaction group?
This variable 9.3 and 10.1 do not see the branches . Most likely it was.
>(I would recommend doing this after importing the pool read-only and
>copy your data off, though).
Понедельник, 30 марта 2015, 11:19 -07:00 от Xin Li :
>-BEGIN PGP SIGNED MESSAGE-
>Hash: SHA512
>
>On 03/30/15 01:32, armonia wrote:
>> Yes, my mistake was probably that I have included data
>> deduplication to see how it works, but not turned it off at the
>> right time. In this case, the machine memory of 4 GB
>>
>> async_destroy - too enabled.
>>
>> That is the conclusion I have deduplication disabled.
>
>Hrm, usually async_destroy should be enough to protect against this
>situation.
>
>Can you try if setting the sysctl variable vfs.zfs.free_max_blocks to
>a limited number, like, 100,000, would make the pool import properly?
> You may have run out of memory because of too much data being free'ed
>in one transaction group?
>
>(I would recommend doing this after importing the pool read-only and
>copy your data off, though).
>
>> How to import a pool of read-only?
>
>zpool import -o readonly poolname.
>
>> Thank you for your response.
>>
>> zpool get all zroot NAME PROPERTY VALUE
>> SOURCE zroot size 230G
>> - zroot capacity 24%
>> - zroot altroot-
>> default zroot health ONLINE
>> - zroot guid 1229884058434432944
>> default zroot version-
>> default zroot bootfs zroot
>> local zroot delegation on
>> default zroot autoreplaceon
>> local zroot cachefile -
>> default zroot failmode wait
>> default zroot listsnapshots on
>> local zroot autoexpand off
>> default zroot dedupditto 0
>> default zroot dedupratio 1.02x
>> - zroot free 174G
>> - zroot allocated 56.1G
>> - zroot readonly off
>> - zroot commentZFS
>> local zroot expandsize 0
>> - zroot freeing0
>> default zroot feature@async_destroy enabled
>> local zroot feature@empty_bpobjactive
>> local zroot feature@lz4_compress active
>> local zroot feature@multi_vdev_crash_dump enabled
>> local zroot feature@spacemap_histogram active
>> local zroot feature@enabled_txgactive
>> local zroot feature@hole_birth active
>> local zroot feature@extensible_dataset enabled
>> local zroot feature@bookmarks enabled
>> local zroot feature@filesystem_limits enabled
>> local
>>
>>
>> ???, 30 ? 2015, 0:36 -07:00 ?? Xin Li
>> < delp...@delphij.net >:
>>
>>
>>
>> On 3/27/15 05:26, armonia wrote:
>>> After importing I press ctrl + t and here's the conclusion:
>>
>>> load: 0.20 cmd: zpool 725 [tx->tx_sync_done_cv] 32.50r 0.00y
>>> 5.59s 0% 6432k
>>
>> Have you ever enabled e.g. dedup on certain dataset(s) and have a
>> lot of files, and the pool don't have 'async destroy' feature
>> enabled? In that case the fastest way to recover, if this took too
>> long, would probably import the pool read-only and copy data to
>> another pool.
>>
>> On -CURRENT you can use dtrace -qn 'zfs-dbgmsg{printf("%s\n",
>> stringof(arg0))}' to see more verbose information, they are not
>> always helpful but will give you better idea on what is going on
>> under the hood .
>>
>> Cheers, ___
>> freebsd-stable@freebsd.org
>> < /compose?To=freebsd%2dsta...@freebsd.org >
