Page 1 of 1

ALERT : Ext4 data corruption trouble on the latest kernels

Posted: 26 October 2012, 10:55
by NicCo
SOLUTION
About kernel 3.5
Fixed RPMS files are ready and available for everybody, here
http://abf.rosalinux.ru/downloads/kerne ... epository/


About kernel 3.6
I prefer to wait for the fix with the incoming 3.6.4
infact right now has been announced 3.6.4 rc1
http://www.kernel.org/pub/linux/kernel/ ... .4-rc1.bz2
ALERT
About the latest kernels > Ext4 data corruption trouble
Read here for Infos: https://lwn.net/Articles/521022/
Stable kernel updates are supposed to be just that — stable. But they are not immune to bugs, as a recent ext4 filesystem problem has shown. In short: ext4 users would be well advised to avoid versions 3.4.14, 3.4.15, 3.5.7, 3.6.2, and 3.6.3; they all contain a patch which can, in some situations, cause filesystem corruption.

The problem, as explained in this note from Ted Ts'o, has to do with how the ext4 journal is managed. In some situations, unmounting the filesystem fails to truncate the journal, leaving stale (but seemingly valid) data there. After a single unmount/remount (or reboot) cycle little harm is done; some old transactions just get replayed unnecessarily. If the filesystem is quickly unmounted again, though, the journal can be left in a corrupted state; that corruption will be helpfully replayed onto the filesystem at the next mount.

Fixes are in the works. The ext4 developers are taking some time, though, to be sure that the problem has been fully understood and completely fixed; there are signs that the bug may have roots far older than the patch that actually caused it to bite people. Once that process is complete, there should be a new round of stable updates (possibly even for 3.5, which is otherwise at end-of-life) and the world will be safe for ext4 users again.

(Thanks are due to LWN reader "nix" who alerted readers in the comments and reported the bug to the ext4 developers).

Update: Ted now thinks that his initial diagnosis was incomplete at best; the problem is not as well understood as it seemed. Stay tuned.
And so, I'm waiting for new kernel 3.5.8 and 3.6.4 to fix the serious problem

In the meanwhile:
Patch for fixing of this error is avaliable: https://lkml.org/lkml/2012/10/23/690

ASAP
Yesterday night I push ABF to rebuild new release version for kernel 3.5.7
not only the mainline serie (now rel. -2) but also for nrjQL (now rel. -69.2)

Here, the changelog of the two versions, mainline and experimental nrjQL
Fixed RPMS files are ready and available for everybody, here
http://abf.rosalinux.ru/downloads/kerne ... epository/

New mainline and official kernels

Code: Select all

%changelog
* Fri Oct 25 2012 Nicolo' Costanza <abitrules@yahoo.it> 3.5.7-2
+ a new rel that should fix this > https://lwn.net/Articles/521022/
- added FX01-fix-serious-progressive-ext4-data-corruption-bug.patch
- ---------------------------------------------------------------------
- Kernel 3.5 for mdv 2010.2, 2011.0, cooker, rosa.lts2012.0, rosa2012.1
- MIB (Mandriva International Backports) - http://mib.pianetalinux.org/
- This is -1 (mainline serie), with official kernel sources and addons,
- instead (-69) will be used for development and experimental flavours
- Yin & Yang (69) release - a very complete but experimental flavours...
- ---------------------------------------------------------------------
New nrjQL advanced and experimental Kernels

Code: Select all

%changelog
* Fri Oct 25 2012 Nicolo' Costanza <abitrules@yahoo.it> 3.5.7-69.2
+ a new rel that should fix this > https://lwn.net/Articles/521022/
- added FX01-fix-serious-progressive-ext4-data-corruption-bug.patch
- ---------------------------------------------------------------------
- Kernel 3.5 for mdv 2010.2, 2011.0, cooker, rosa.lts2012.0, rosa2012.1
- MIB (Mandriva International Backports) - http://mib.pianetalinux.org/
- The rel -1 (mainline serie), with official kernel sources and addons,
- instead (-69) will be used for development and experimental flavours
- Yin & Yang (69) release - a very complete but experimental flavours...
- ---------------------------------------------------------------------

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 26 October 2012, 13:42
by oiram73
A me si comporta come il 3.6.x:
3.5.7-2 non parte proprio!

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 26 October 2012, 14:11
by NicCo
oiram73 wrote:A me si comporta come il 3.6.x:
3.5.7-2 non parte proprio!
Ho notato che su RM2012 alcuni nuovi kernel, superiori alla serie 3.4, hanno difficoltà a partire.
Da una analisi superficiale, mi è sembrato di scorgere una incompatibilità tra la versione di udev
presente e i nuovi kernel.
Sul mio PC di sviluppo, con RM2012 i kernel 3.6 neanche partono proprio per questo motivo...
Su altre RM 2012 vanno perfettamente, dipenderà probabilmente dall'hardware...
Se invece li provo con ROSA 2012.1 è tutto OK
Quello che fa sorridere è che se provo MDV 2010 è tutto OK

P.S. proprio in questo momento ho mandato al rebuild, il nuovo kernel 3.6.4, anche se solo rc1
ho aggiunto la patch per l'ext4 e sto buildando solo i flavour nrj-desktop, nrj-laptop e relativi pae

Cosi quelli che hanno il 3.6.3 e hanno qualche timore che possa loro avvenire una perdita di dati,
potranno subire aggiornare al kernel temporaneo 3.6.4-0-rc1.1

Non voglio mai che si dica, ho perso i dati per incuria di NicCo... ;)

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 26 October 2012, 15:32
by NicCo
Build for new Kernel 3.6.4 is near to completing into ABF:
I can see the list right now, is grabbed from the build log

I'll write you the temporary repository folders to install it,
waiting when all build will be completed in some hours...

Code: Select all

Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/build/rpmbuild/BUILDROOT/kernel-3.6.4-0.rc1.1-mib2011.0.x86_64-buildroot
Wrote: /home/build/rpmbuild/SRPMS/kernel-3.6.4-0.rc1.1.src.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-desktop-3.6.4-0.rc1.1mib-1-1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-desktop-devel-3.6.4-0.rc1.1mib-1-1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-desktop-latest-3.6.4-0.rc1.1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-desktop-devel-latest-3.6.4-0.rc1.1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-laptop-3.6.4-0.rc1.1mib-1-1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-laptop-devel-3.6.4-0.rc1.1mib-1-1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-laptop-latest-3.6.4-0.rc1.1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/x86_64/kernel-nrj-laptop-devel-latest-3.6.4-0.rc1.1-mib2011.0.x86_64.rpm
Wrote: /home/build/rpmbuild/RPMS/noarch/kernel-source-3.6.4-0.rc1.1mib-1-1-mib2011.0.noarch.rpm
Wrote: /home/build/rpmbuild/RPMS/noarch/kernel-source-latest-3.6.4-0.rc1.1-mib2011.0.noarch.rpm
Wrote: /home/build/rpmbuild/RPMS/noarch/kernel-doc-3.6.4-0.rc1.1-mib2011.0.noarch.rpm
Executing(%clean): /bin/sh -e /var/tmp/rpm-tmp.97362
+ umask 022
+ cd /home/build/rpmbuild/BUILD
+ cd kernel-x86_64
+ /bin/rm -rf /home/build/rpmbuild/BUILDROOT/kernel-3.6.4-0.rc1.1-mib2011.0.x86_64-buildroot
here the changelog

Code: Select all

%changelog

* Fri Oct 26 2012 Nicolo' Costanza <abitrules@yahoo.it> 3.6.4-rc1
+ update to 3.6.4-rc1
+ rc release that should fix this > https://lwn.net/Articles/521022/
- added FX01-fix-serious-progressive-ext4-data-corruption-bug.patch
- ---------------------------------------------------------------------
- Kernel 3.6 for mdv 2010.2, 2011.0, cooker, rosa.lts2012.0, rosa2012.1
- MIB (Mandriva International Backports) - http://mib.pianetalinux.org/
- This is -1 (mainline serie), with official kernel sources and addons,
- instead (-69) will be used for development and experimental flavours
- Yin & Yang (69) release - a very complete but experimental flavours...
- ---------------------------------------------------------------------

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 26 October 2012, 15:43
by NicCo
Kernel 3.6.4 rc1 for ROSA and Mandriva Linux
For the users with kernel 3.6.3, if they wish to install 3.6.4 rc1 with the patch to fix the ext4 bug, here the files
to make the builds complete faster, I enabled only nrj-desktop, nrj-laptop and their relative 32bit pae versions
All users of the 3.6 kernel series must upgrade.
I do not want to be told never around: "I have lost my data due to some carelessness of NicCo ..." :) ;)

Naturally this is only a temporary solution, until the final and official kernel 3.6.4 will be availalble (not rc)


Mandriva 2011 - x86_64
http://abf.rosalinux.ru/downloads/kerne ... 7367/RPMS/

ROSA RM2012 - x86_64
http://abf.rosalinux.ru/downloads/kerne ... 7366/RPMS/

ROSA 2012.1 - x86_64
http://abf.rosalinux.ru/downloads/kerne ... 7368/RPMS/



Mandriva 2011 - i586 (build in progress, it needs a couple of hours to completing...)
http://abf.rosalinux.ru/downloads/kerne ... 7364/RPMS/

ROSA RM2012 - i586 build in progress, it needs a couple of hours to completing...)
http://abf.rosalinux.ru/downloads/kerne ... 7363/RPMS/

ROSA 2012.1 - i586 build in progress, it needs a couple of hours to completing...)
http://abf.rosalinux.ru/downloads/kerne ... 7365/RPMS/


My little personal testing
I installed in my ROSA 2012.1 - x86_64 - It works fine!

Code: Select all

[piero@localhost ~]$ uname -a && cat /etc/product.id
Linux localhost.localdomain 3.6.4-nrj-desktop-0.rc1.1rosa #1 SMP PREEMPT Fri Oct 26 12:37:51 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
vendor=ROSA,distribution=ROSA Linux,type=Basic,version=2012.1,branch=Official,release=1,arch=x86_64,product=EE

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 9:29
by promeneur
hello

3.5.7 is a dangerous kernel and you didn't delete it in the packet source of mandriva 2010.2 i586

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 9:38
by NicCo
promeneur wrote:hello

3.5.7 is a dangerous kernel and you didn't delete it in the packet source of mandriva 2010.2 i586
We will replace it soon with the fixed 3.5.7-2

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 14:11
by Alex-G
NicCo wrote:
oiram73 wrote:A me si comporta come il 3.6.x:
3.5.7-2 non parte proprio!
Ho notato che su RM2012 alcuni nuovi kernel, superiori alla serie 3.4, hanno difficoltà a partire.
Da una analisi superficiale, mi è sembrato di scorgere una incompatibilità tra la versione di udev
presente e i nuovi kernel.
Sul mio PC di sviluppo, con RM2012 i kernel 3.6 neanche partono proprio per questo motivo...
Su altre RM 2012 vanno perfettamente, dipenderà probabilmente dall'hardware...
Se invece li provo con ROSA 2012.1 è tutto OK
Quello che fa sorridere è che se provo MDV 2010 è tutto OK
ed è anche per questo che per il momento rimango *fedele* a questa versione che mi ha sempre dato pochi problemi, nessuno di questi gravi e tutti risolvibili e *MOLTE* soddisfzioni; prima di passare ad altrovoglio pianificare bene il passaggio mantenendo le applicazioni che mi servono...
P.S. proprio in questo momento ho mandato al rebuild, il nuovo kernel 3.6.4, anche se solo rc1 ho aggiunto la patch per l'ext4 e sto buildando solo i flavour nrj-desktop, nrj-laptop e relativi pae

Cosi quelli che hanno il 3.6.3 e hanno qualche timore che possa loro avvenire una perdita di dati,
potranno subire aggiornare al kernel temporaneo 3.6.4-0-rc1.1
Non voglio mai che si dica, ho perso i dati per incuria di NicCo... ;)
Ho un netbook Atom e per il momento vado con Mdv2010.2+ aggiornamento 2010.3+kernel 3.5.4 +Xorg 7.7, Xserver 1.13 and Mesa 9.0 e tutto fila liscio....

Visti i recenti problemi dei kernel con Ext4 ho lasciato perdere ulteriori aggiornamenti in attesa di migliori notizie,
il 3.6.4 è sparito dagli archivi 2010.2
http://mib.pianetalinux.org/MIB/2010.2/ ... ts/kernels
era difettoso anche quello? Mi conviene il 3.6.1 (che però non ha la versione per Atom) o meglio attendere?
Grazie

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 16:28
by promeneur
i have just pb with ext4 partition with kernel 3.4.11

happilyit's not an important partition

kernel 3.4.11 is not in kernel list of kernels not to use

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 16:55
by promeneur
3.4.11 is not concerned with my partition pb

i had a connector pb with a disk

i fixed it.

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 18:07
by promeneur
are you aware of this new corruption pb about ext4 ?

see

http://www.phoronix.com/scan.php?page=n ... px=MTIxOTU

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 1 November 2012, 18:53
by NicCo
promeneur wrote:are you aware of this new corruption pb about ext4 ?

see

http://www.phoronix.com/scan.php?page=n ... px=MTIxOTU
Thank you for reporting the issue,
so let's stay alerted, but now there is nothing I can do for...


affected Ubuntu kernel version is 3.5.x serie, Ubuntu uses strange numbering releases, not the official
and Ubuntu kernel we know that is far different than a kernel for ROSA and Mandriva, especially "nrj" one...

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 2 November 2012, 17:09
by NicCo
UPDATE
In the incoming > Kernel 3.6.5 QL
there will be a new important fix,

an upstream patch whose name:
'ext4: fix unjournaled inode bitmap modification'

finally should fix, in the right way, the data corruption trouble

Code: Select all

%changelog

* Fri Nov 02 2012 Nicolo' Costanza <abitrules@yahoo.it> 3.6.5-69
+ update to 3.6.5-69
- updated QL patches
- add FX01_fs-ext4-fix-unjournaled-inode-bitmap-modification.patch
- drop FX01_fix-serious-progressive-ext4-data-corruption-bug.patch
- add video4linux vloopback support
- http://www.lavrsen.dk/foswiki/bin/view/Motion/VideoFourLinuxLoopbackDevice
- ---------------------------------------------------------------------
- Kernel 3.6 for mdv 2010.2, 2011.0, cooker, rosa.lts2012.0, rosa2012.1
- MIB (Mandriva International Backports) - http://mib.pianetalinux.org/
- The rel -1 (mainline serie), with official kernel sources and addons,
- instead (-69) will be used for development and experimental flavours
- Yin & Yang (69) release - a very complete but experimental flavours...
- ---------------------------------------------------------------------

* Sun Oct 28 2012 Nicolo' Costanza <abitrules@yahoo.it> 3.6.4-69
+ update to 3.6.4-69
+ about the ext4 problem discussed > https://lwn.net/Articles/521022/
- added FX01-fix-serious-progressive-ext4-data-corruption-bug.patch
- ---------------------------------------------------------------------
- Kernel 3.6 for mdv 2010.2, 2011.0, cooker, rosa.lts2012.0, rosa2012.1
- MIB (Mandriva International Backports) - http://mib.pianetalinux.org/
- The rel -1 (mainline serie), with official kernel sources and addons,
- instead (-69) will be used for development and experimental flavours
- Yin & Yang (69) release - a very complete but experimental flavours...
- ---------------------------------------------------------------------
The same patch should be inserted as default in next stable 3.6.6


Denis Koryavov in ROSA ML specified that the error is not so frequent, so it's better don't worry so much ;)
I agree with him! Infact, there have been many alerts like this last one in the past, but damage was very very unfrequent
Denis Koryavov wrote:
The bug with ext4 is more theoretical than real and appears only on some configurations
with the some mount options (journal_checksum and journal_async_commit).

https://plus.google.com/u/0/11709138045 ... 5a1eHxUzSh

Here is my files from a first local build

Code: Select all

$ time rpmbuild --bb /home/utente/rpmbuild/SPECS/kernel-3.6.5-69.spec
..............................................................................................
Checking for unpackaged file(s): /usr/lib/rpm/check-files /home/utente/rpmbuild/BUILDROOT/kernel-3.6.5-69-rosa.lts2012.0.x86_64-buildroot
Wrote: /home/utente/rpmbuild/RPMS/x86_64/kernel-nrjQL-desktop-3.6.5-69rosa.lts-1-1-rosa.lts2012.0.x86_64.rpm
Wrote: /home/utente/rpmbuild/RPMS/x86_64/kernel-nrjQL-desktop-devel-3.6.5-69rosa.lts-1-1-rosa.lts2012.0.x86_64.rpm
Wrote: /home/utente/rpmbuild/RPMS/x86_64/kernel-nrjQL-desktop-latest-3.6.5-69-rosa.lts2012.0.x86_64.rpm
Wrote: /home/utente/rpmbuild/RPMS/x86_64/kernel-nrjQL-desktop-devel-latest-3.6.5-69-rosa.lts2012.0.x86_64.rpm
Wrote: /home/utente/rpmbuild/RPMS/noarch/kernel-source-3.6.5-69rosa.lts-1-1-rosa.lts2012.0.noarch.rpm
Wrote: /home/utente/rpmbuild/RPMS/noarch/kernel-source-latest-3.6.5-69-rosa.lts2012.0.noarch.rpm
Wrote: /home/utente/rpmbuild/RPMS/noarch/kernel-doc-3.6.5-69-rosa.lts2012.0.noarch.rpm
Wrote: /home/utente/rpmbuild/RPMS/x86_64/perf-3.6.5-69-rosa.lts2012.0.x86_64.rpm
Wrote: /home/utente/rpmbuild/RPMS/x86_64/cpupower-3.6.5-69-rosa.lts2012.0.x86_64.rpm
Wrote: /home/utente/rpmbuild/RPMS/x86_64/cpupower-devel-3.6.5-69-rosa.lts2012.0.x86_64.rpm
Esecuzione(%clean): /bin/sh -e /var/tmp/rpm-tmp.84077
+ umask 022
+ cd /home/utente/rpmbuild/BUILD
+ cd kernel-x86_64
+ /bin/rm -rf /home/utente/rpmbuild/BUILDROOT/kernel-3.6.5-69-rosa.lts2012.0.x86_64-buildroot

real    76m9.871s
user    77m49.395s
sys     10m5.798s
utente /media/2TB/mirror/mib/rosa2012lts/32/experts $ 
bye, NicCo

Re: ALERT : Ext4 data corruption trouble on the latest kerne

Posted: 3 November 2012, 10:57
by NicCo
In the new kernel versions that would be released in some days

Kernel 3.4.18 rc1
http://www.mail-archive.com/linux-kerne ... 34156.html

Kernel 3.6.6 rc1
https://lkml.org/lkml/2012/11/2/432
http://comments.gmane.org/gmane.linux.kernel/1387691


You can see that the ext4 patch has been accepted

Eric Sandeen <sandeen@redhat.com>
ext4: fix unjournaled inode bitmap modification


so our latest kernel 3.6.5 should be immune to the trouble bug,

for who feels confortable with kernel 3.5, a new version of 3.5.7
will be prepared in the next week, and will contain such patch...