panic when mounting 8To ntfs partition

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

panic when mounting 8To ntfs partition

Landry Breuil-5
Hi,

i know we don't really have an ntfs maintainer, and that nobody should
use ntfs, but interoperability...

i have a 8Tb 'seagate backup plus hub' appearing as:

uhub8 at uhub1 port 5 configuration 1 interface 0 "Seagate Backup+ Hub" rev 2.10/48.85 addr 2
umass0 at uhub8 port 1 configuration 1 interface 0 "Seagate Backup+ Hub BK" rev 2.10/1.00 addr 3
umass0: using SCSI over Bulk-Only
scsibus4 at umass0: 2 targets, initiator 0
sd2 at scsibus4 targ 1 lun 0: <Seagate, Backup+ Hub BK, D781> SCSI4 0/direct fixed

which just has a huge ntfs partition:

# /dev/rsd2c:
type: SCSI
disk: SCSI disk
label: Backup+ Hub BK  
duid: 0000000000000000
flags:
bytes/sector: 512
sectors/track: 63
tracks/cylinder: 255
sectors/cylinder: 16065
cylinders: 972801
total sectors: 15628053167
boundstart: 0
boundend: 15628053167
drivedata: 0

16 partitions:
#                size           offset  fstype [fsize bsize   cpg]
  c:      15628053167                0  unused                    
  i:           262144               34 unknown                    # /mnt/sd2
  j:      15627788288           264192   MSDOS              

(with -pG)
total sectors: 15628053167 # total bytes: 7452.0G
  c:          7452.0G                0  unused                    
  i:             0.1G               34 unknown                    # /mnt/sd2
  j:          7451.9G           264192   MSDOS      

Trying to mount it (dev/sd2j, of course) immediately panics the kernel
(hand-written):

panic: out of space in kmem_map
panic
malloc
ntfs_calccfree
ntfs_mountfs
ntfs_mount
sys_mount
syscall

I suppose pointing at
https://github.com/openbsd/src/blob/master/sys/ntfs/ntfs_vfsops.c#L567

So, what can be done to 1) avoid the panic and 2) eventually find a way to
support those partitions sizes ?

Landry

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: panic when mounting 8To ntfs partition

Landry Breuil-5
On Wed, Mar 01, 2017 at 10:28:41AM +0100, Landry Breuil wrote:

> Hi,
>
> i know we don't really have an ntfs maintainer, and that nobody should
> use ntfs, but interoperability...
>
> i have a 8Tb 'seagate backup plus hub' appearing as:
>
> uhub8 at uhub1 port 5 configuration 1 interface 0 "Seagate Backup+ Hub" rev 2.10/48.85 addr 2
> umass0 at uhub8 port 1 configuration 1 interface 0 "Seagate Backup+ Hub BK" rev 2.10/1.00 addr 3
> umass0: using SCSI over Bulk-Only
> scsibus4 at umass0: 2 targets, initiator 0
> sd2 at scsibus4 targ 1 lun 0: <Seagate, Backup+ Hub BK, D781> SCSI4 0/direct fixed
>
> which just has a huge ntfs partition:
>
> # /dev/rsd2c:
> type: SCSI
> disk: SCSI disk
> label: Backup+ Hub BK  
> duid: 0000000000000000
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 972801
> total sectors: 15628053167
> boundstart: 0
> boundend: 15628053167
> drivedata: 0
>
> 16 partitions:
> #                size           offset  fstype [fsize bsize   cpg]
>   c:      15628053167                0  unused                    
>   i:           262144               34 unknown                    # /mnt/sd2
>   j:      15627788288           264192   MSDOS              
>
> (with -pG)
> total sectors: 15628053167 # total bytes: 7452.0G
>   c:          7452.0G                0  unused                    
>   i:             0.1G               34 unknown                    # /mnt/sd2
>   j:          7451.9G           264192   MSDOS      
>
> Trying to mount it (dev/sd2j, of course) immediately panics the kernel
> (hand-written):
>
> panic: out of space in kmem_map
> panic
> malloc
> ntfs_calccfree
> ntfs_mountfs
> ntfs_mount
> sys_mount
> syscall

And since it's reproductible, here's another one with offsets this time:

panic: malloc: out of space in kmem_map
panic()
malloc+0x4b1
ntfs_calccfree+0x3b
ntfs_mountfs+0x3f9
ntfs_mount+0x21c
sys_mount+0x271
syscall+0x27b
syscall number 21
end trace frame 0x7f7ffffc3901 count:7
0x1a4e3d60109a

That's with a self-built kernel from
OpenBSD 6.0-current (GENERIC.MP) #4: Mon Feb 13 08:58:42 CET 2017

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: panic when mounting 8To ntfs partition

Jeremie Courreges-Anglas-2
In reply to this post by Landry Breuil-5
Landry Breuil <[hidden email]> writes:

> Hi,
>
> i know we don't really have an ntfs maintainer, and that nobody should
> use ntfs, but interoperability...
>
> i have a 8Tb 'seagate backup plus hub' appearing as:
>
> uhub8 at uhub1 port 5 configuration 1 interface 0 "Seagate Backup+ Hub" rev 2.10/48.85 addr 2
> umass0 at uhub8 port 1 configuration 1 interface 0 "Seagate Backup+ Hub BK" rev 2.10/1.00 addr 3
> umass0: using SCSI over Bulk-Only
> scsibus4 at umass0: 2 targets, initiator 0
> sd2 at scsibus4 targ 1 lun 0: <Seagate, Backup+ Hub BK, D781> SCSI4 0/direct fixed
>
> which just has a huge ntfs partition:
>
> # /dev/rsd2c:
> type: SCSI
> disk: SCSI disk
> label: Backup+ Hub BK  
> duid: 0000000000000000
> flags:
> bytes/sector: 512
> sectors/track: 63
> tracks/cylinder: 255
> sectors/cylinder: 16065
> cylinders: 972801
> total sectors: 15628053167
> boundstart: 0
> boundend: 15628053167
> drivedata: 0
>
> 16 partitions:
> #                size           offset  fstype [fsize bsize   cpg]
>   c:      15628053167                0  unused                    
>   i:           262144               34 unknown                    # /mnt/sd2
>   j:      15627788288           264192   MSDOS              
>
> (with -pG)
> total sectors: 15628053167 # total bytes: 7452.0G
>   c:          7452.0G                0  unused                    
>   i:             0.1G               34 unknown                    # /mnt/sd2
>   j:          7451.9G           264192   MSDOS      
>
> Trying to mount it (dev/sd2j, of course) immediately panics the kernel
> (hand-written):
>
> panic: out of space in kmem_map
> panic
> malloc
> ntfs_calccfree
> ntfs_mountfs
> ntfs_mount
> sys_mount
> syscall
>
> I suppose pointing at
> https://github.com/openbsd/src/blob/master/sys/ntfs/ntfs_vfsops.c#L567
>
> So, what can be done to 1) avoid the panic and 2) eventually find a way to
> support those partitions sizes ?

I guess that we should not allocate memory for the whole bitmap file,
but instead split it in chunks.

I have only tested with dummy ntfs volumes on vnd devices, but this
fixes landry's panic and allows him to browse his NTFS filesystem.

Concerns:
- what would be a better chunk size than 1MB?
- uint64_t vs off_t: add more tests

Input welcome.


Index: ntfs/ntfs_vfsops.c
===================================================================
RCS file: /d/cvs/src/sys/ntfs/ntfs_vfsops.c,v
retrieving revision 1.55
diff -u -p -r1.55 ntfs_vfsops.c
--- ntfs/ntfs_vfsops.c 7 Sep 2016 17:30:12 -0000 1.55
+++ ntfs/ntfs_vfsops.c 3 Mar 2017 17:52:18 -0000
@@ -558,22 +558,35 @@ ntfs_calccfree(struct ntfsmount *ntmp, c
  u_int8_t *tmp;
  int j, error;
  cn_t cfree = 0;
- size_t bmsize, i;
+ uint64_t bmsize, offset;
+ size_t chunksize, i;
 
  vp = ntmp->ntm_sysvn[NTFS_BITMAPINO];
 
  bmsize = VTOF(vp)->f_size;
 
- tmp = malloc(bmsize, M_TEMP, M_WAITOK);
+ if (bmsize > 1024 * 1024)
+ chunksize = 1024 * 1024;
+ else
+ chunksize = bmsize;
+
+ tmp = malloc(chunksize, M_TEMP, M_WAITOK);
+
+ for (offset = 0; offset < bmsize; offset += chunksize) {
+ if (chunksize > bmsize - offset)
+ chunksize = bmsize - offset;
+
+ error = ntfs_readattr(ntmp, VTONT(vp), NTFS_A_DATA, NULL,
+    offset, chunksize, tmp, NULL);
+ if (error)
+ goto out;
+
+ for (i = 0; i < chunksize; i++)
+ for (j = 0; j < 8; j++)
+ if (~tmp[i] & (1 << j))
+ cfree++;
+ }
 
- error = ntfs_readattr(ntmp, VTONT(vp), NTFS_A_DATA, NULL,
-       0, bmsize, tmp, NULL);
- if (error)
- goto out;
-
- for(i=0;i<bmsize;i++)
- for(j=0;j<8;j++)
- if(~tmp[i] & (1 << j)) cfree++;
  *cfreep = cfree;
 
     out:


--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: panic when mounting 8To ntfs partition

Jeremie Courreges-Anglas-2
Jeremie Courreges-Anglas <[hidden email]> writes:

> Landry Breuil <[hidden email]> writes:
>
>> Hi,
>>
>> i know we don't really have an ntfs maintainer, and that nobody should
>> use ntfs, but interoperability...
>>
>> i have a 8Tb 'seagate backup plus hub' appearing as:
>>
>> uhub8 at uhub1 port 5 configuration 1 interface 0 "Seagate Backup+ Hub" rev 2.10/48.85 addr 2
>> umass0 at uhub8 port 1 configuration 1 interface 0 "Seagate Backup+ Hub BK" rev 2.10/1.00 addr 3
>> umass0: using SCSI over Bulk-Only
>> scsibus4 at umass0: 2 targets, initiator 0
>> sd2 at scsibus4 targ 1 lun 0: <Seagate, Backup+ Hub BK, D781> SCSI4 0/direct fixed
>>
>> which just has a huge ntfs partition:
>>
>> # /dev/rsd2c:
>> type: SCSI
>> disk: SCSI disk
>> label: Backup+ Hub BK  
>> duid: 0000000000000000
>> flags:
>> bytes/sector: 512
>> sectors/track: 63
>> tracks/cylinder: 255
>> sectors/cylinder: 16065
>> cylinders: 972801
>> total sectors: 15628053167
>> boundstart: 0
>> boundend: 15628053167
>> drivedata: 0
>>
>> 16 partitions:
>> #                size           offset  fstype [fsize bsize   cpg]
>>   c:      15628053167                0  unused                    
>>   i:           262144               34 unknown                    # /mnt/sd2
>>   j:      15627788288           264192   MSDOS              
>>
>> (with -pG)
>> total sectors: 15628053167 # total bytes: 7452.0G
>>   c:          7452.0G                0  unused                    
>>   i:             0.1G               34 unknown                    # /mnt/sd2
>>   j:          7451.9G           264192   MSDOS      
>>
>> Trying to mount it (dev/sd2j, of course) immediately panics the kernel
>> (hand-written):
>>
>> panic: out of space in kmem_map
>> panic
>> malloc
>> ntfs_calccfree
>> ntfs_mountfs
>> ntfs_mount
>> sys_mount
>> syscall
>>
>> I suppose pointing at
>> https://github.com/openbsd/src/blob/master/sys/ntfs/ntfs_vfsops.c#L567
>>
>> So, what can be done to 1) avoid the panic and 2) eventually find a way to
>> support those partitions sizes ?
>
> I guess that we should not allocate memory for the whole bitmap file,
> but instead split it in chunks.
>
> I have only tested with dummy ntfs volumes on vnd devices, but this
> fixes landry's panic and allows him to browse his NTFS filesystem.
>
> Concerns:
> - what would be a better chunk size than 1MB?
> - uint64_t vs off_t: add more tests
>
> Input welcome.

No opinion on that one?

> Index: ntfs/ntfs_vfsops.c
> ===================================================================
> RCS file: /d/cvs/src/sys/ntfs/ntfs_vfsops.c,v
> retrieving revision 1.55
> diff -u -p -r1.55 ntfs_vfsops.c
> --- ntfs/ntfs_vfsops.c 7 Sep 2016 17:30:12 -0000 1.55
> +++ ntfs/ntfs_vfsops.c 3 Mar 2017 17:52:18 -0000
> @@ -558,22 +558,35 @@ ntfs_calccfree(struct ntfsmount *ntmp, c
>   u_int8_t *tmp;
>   int j, error;
>   cn_t cfree = 0;
> - size_t bmsize, i;
> + uint64_t bmsize, offset;
> + size_t chunksize, i;
>  
>   vp = ntmp->ntm_sysvn[NTFS_BITMAPINO];
>  
>   bmsize = VTOF(vp)->f_size;
>  
> - tmp = malloc(bmsize, M_TEMP, M_WAITOK);
> + if (bmsize > 1024 * 1024)
> + chunksize = 1024 * 1024;
> + else
> + chunksize = bmsize;
> +
> + tmp = malloc(chunksize, M_TEMP, M_WAITOK);
> +
> + for (offset = 0; offset < bmsize; offset += chunksize) {
> + if (chunksize > bmsize - offset)
> + chunksize = bmsize - offset;
> +
> + error = ntfs_readattr(ntmp, VTONT(vp), NTFS_A_DATA, NULL,
> +    offset, chunksize, tmp, NULL);
> + if (error)
> + goto out;
> +
> + for (i = 0; i < chunksize; i++)
> + for (j = 0; j < 8; j++)
> + if (~tmp[i] & (1 << j))
> + cfree++;
> + }
>  
> - error = ntfs_readattr(ntmp, VTONT(vp), NTFS_A_DATA, NULL,
> -       0, bmsize, tmp, NULL);
> - if (error)
> - goto out;
> -
> - for(i=0;i<bmsize;i++)
> - for(j=0;j<8;j++)
> - if(~tmp[i] & (1 << j)) cfree++;
>   *cfreep = cfree;
>  
>      out:

--
jca | PGP : 0x1524E7EE / 5135 92C1 AD36 5293 2BDF  DDCC 0DFA 74AE 1524 E7EE

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: panic when mounting 8To ntfs partition

Landry Breuil-5
On Tue, Mar 14, 2017 at 11:58:23AM +0100, Jeremie Courreges-Anglas wrote:

> Jeremie Courreges-Anglas <[hidden email]> writes:
>
> > Landry Breuil <[hidden email]> writes:
> >
> >> Hi,
> >>
> >> i know we don't really have an ntfs maintainer, and that nobody should
> >> use ntfs, but interoperability...
> >>
> >> i have a 8Tb 'seagate backup plus hub' appearing as:
> >>
> >> uhub8 at uhub1 port 5 configuration 1 interface 0 "Seagate Backup+ Hub" rev 2.10/48.85 addr 2
> >> umass0 at uhub8 port 1 configuration 1 interface 0 "Seagate Backup+ Hub BK" rev 2.10/1.00 addr 3
> >> umass0: using SCSI over Bulk-Only
> >> scsibus4 at umass0: 2 targets, initiator 0
> >> sd2 at scsibus4 targ 1 lun 0: <Seagate, Backup+ Hub BK, D781> SCSI4 0/direct fixed
> >>
> >> which just has a huge ntfs partition:
> >>
> >> # /dev/rsd2c:
> >> type: SCSI
> >> disk: SCSI disk
> >> label: Backup+ Hub BK  
> >> duid: 0000000000000000
> >> flags:
> >> bytes/sector: 512
> >> sectors/track: 63
> >> tracks/cylinder: 255
> >> sectors/cylinder: 16065
> >> cylinders: 972801
> >> total sectors: 15628053167
> >> boundstart: 0
> >> boundend: 15628053167
> >> drivedata: 0
> >>
> >> 16 partitions:
> >> #                size           offset  fstype [fsize bsize   cpg]
> >>   c:      15628053167                0  unused                    
> >>   i:           262144               34 unknown                    # /mnt/sd2
> >>   j:      15627788288           264192   MSDOS              
> >>
> >> (with -pG)
> >> total sectors: 15628053167 # total bytes: 7452.0G
> >>   c:          7452.0G                0  unused                    
> >>   i:             0.1G               34 unknown                    # /mnt/sd2
> >>   j:          7451.9G           264192   MSDOS      
> >>
> >> Trying to mount it (dev/sd2j, of course) immediately panics the kernel
> >> (hand-written):
> >>
> >> panic: out of space in kmem_map
> >> panic
> >> malloc
> >> ntfs_calccfree
> >> ntfs_mountfs
> >> ntfs_mount
> >> sys_mount
> >> syscall
> >>
> >> I suppose pointing at
> >> https://github.com/openbsd/src/blob/master/sys/ntfs/ntfs_vfsops.c#L567
> >>
> >> So, what can be done to 1) avoid the panic and 2) eventually find a way to
> >> support those partitions sizes ?
> >
> > I guess that we should not allocate memory for the whole bitmap file,
> > but instead split it in chunks.
> >
> > I have only tested with dummy ntfs volumes on vnd devices, but this
> > fixes landry's panic and allows him to browse his NTFS filesystem.
> >
> > Concerns:
> > - what would be a better chunk size than 1MB?
> > - uint64_t vs off_t: add more tests
> >
> > Input welcome.
>
> No opinion on that one?

I thought i had replied to the list but it seems i only replied to
jeremie.. with this diff, i've been able to mount and browse the ntfs
partition on the 8tb disk without issue. Mounting is slow, but i guess
that's expected.

$doas time mount -t ntfs /dev/sd2j /mnt/sd2
50.33 real         0.00 user         4.29 sys
$df -h /mnt/sd2
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd2j      7.3T    5.9T    1.4T    81%    /mnt/sd2

So even if that's not an ok (since i have no authority in this area),
that's a yay - at least it fixes the panic and restores the
basic functionality - thanks!

Landry

Loading...