How to break the httpd's 4G file size limit?

classic Classic list List threaded Threaded
13 messages Options
Reply | Threaded
Open this post in threaded view
|

How to break the httpd's 4G file size limit?

John Wong
I want to set up the web server to share file, but i know apache-1.3.x
(which is openbsd default httpd) had the 4G file size limit, can i break
this limit?

Thank you.

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

daniel.ramaley (Bugzilla)
On 2009-03-10 at 14:34:30, you wrote:
>I want to set up the web server to share file, but i know apache-1.3.x
>(which is openbsd default httpd) had the 4G file size limit, can i
> break this limit?

I don't know the correct answer to this question, but i thought of a
possible work-around in the event the answer is "no".

Could you write a CGI program that serves the file? I don't know where
Apache's 4 GB limit is. But if the limitation is in how Apache accesses
the filesystem and not in how it manages network connections, then
maybe a program that is capable of reading large files could get around
the limit. This would most likely require mangling the URLs to the
large files to really point to the program, unless you can do some
mod_rewrite magic.

------------------------------------------------------------------------
Dan Ramaley                            Dial Center 118, Drake University
Network Programmer/Analyst             2407 Carpenter Ave
+1 515 271-4540                        Des Moines IA 50311 USA

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Alexey Suslikov
In reply to this post by John Wong
Daniel A. Ramaley wrote:

> On 2009-03-10 at 14:34:30, you wrote:
> >I want to set up the web server to share file, but i know apache-1.3.x
> >(which is openbsd default httpd) had the 4G file size limit, can i
> > break this limit?
>
> I don't know the correct answer to this question, but i thought of a
> possible work-around in the event the answer is "no".
>
> Could you write a CGI program that serves the file? I don't know where
> Apache's 4 GB limit is. But if the limitation is in how Apache accesses
> the filesystem and not in how it manages network connections, then
> maybe a program that is capable of reading large files could get around
> the limit. This would most likely require mangling the URLs to the
> large files to really point to the program, unless you can do some

The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).

Stock ftpd also has mentioned limitation (try to REST a file beyond 2Gb
offset).

Using any 64-bit platform will solve the problem due to bigger off_t, but
if it's not a case, one can use Apache 2.2.x from ports.

Alexey

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Otto Moerbeek
On Wed, Mar 11, 2009 at 05:37:14PM +0200, Alexey Suslikov wrote:

> Daniel A. Ramaley wrote:
>
> > On 2009-03-10 at 14:34:30, you wrote:
> > >I want to set up the web server to share file, but i know apache-1.3.x
> > >(which is openbsd default httpd) had the 4G file size limit, can i
> > > break this limit?
> >
> > I don't know the correct answer to this question, but i thought of a
> > possible work-around in the event the answer is "no".
> >
> > Could you write a CGI program that serves the file? I don't know where
> > Apache's 4 GB limit is. But if the limitation is in how Apache accesses
> > the filesystem and not in how it manages network connections, then
> > maybe a program that is capable of reading large files could get around
> > the limit. This would most likely require mangling the URLs to the
> > large files to really point to the program, unless you can do some
>
> The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).
>
> Stock ftpd also has mentioned limitation (try to REST a file beyond 2Gb
> offset).
>
> Using any 64-bit platform will solve the problem due to bigger off_t, but
> if it's not a case, one can use Apache 2.2.x from ports.
>
> Alexey

This is not correct. off_t is 64-bit on all platforms.

        -Otto

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Alexander Yurchenko-3
In reply to this post by Alexey Suslikov
On Wed, Mar 11, 2009 at 05:37:14PM +0200, Alexey Suslikov wrote:
> The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).

huh?

[grange@nohead tmp]$ cat x.c  
#include <sys/types.h>
#include <stdio.h>

int main(void)
{
        printf("size of off_t is %u bits\n", 8 * sizeof(off_t));
}
[grange@nohead tmp]$ cc x.c  
[grange@nohead tmp]$ ./a.out                                                  
size of off_t is 64 bits
[grange@nohead tmp]$ arch -s  
i386

>
> Stock ftpd also has mentioned limitation (try to REST a file beyond 2Gb
> offset).
>
> Using any 64-bit platform will solve the problem due to bigger off_t, but
> if it's not a case, one can use Apache 2.2.x from ports.
>
> Alexey

--
   Alexander Yurchenko

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Ted Unangst-2
In reply to this post by Alexey Suslikov
On Wed, Mar 11, 2009 at 11:37 AM, Alexey Suslikov
<[hidden email]> wrote:
> The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).

off_t is always 64-bit.

> Stock ftpd also has mentioned limitation (try to REST a file beyond 2Gb
> offset).
>
> Using any 64-bit platform will solve the problem due to bigger off_t, but
> if it's not a case, one can use Apache 2.2.x from ports.

If using a 64-bit machine fixes it, then the type in question is not off_t.

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Alexey Suslikov
On Wed, Mar 11, 2009 at 17:56, Ted Unangst <[hidden email]> wrote:

> On Wed, Mar 11, 2009 at 11:37 AM, Alexey Suslikov
> <[hidden email]> wrote:
>> The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).
>
> off_t is always 64-bit.
>
>> Stock ftpd also has mentioned limitation (try to REST a file beyond 2Gb
>> offset).
>>
>> Using any 64-bit platform will solve the problem due to bigger off_t, but
>> if it's not a case, one can use Apache 2.2.x from ports.
>
> If using a 64-bit machine fixes it, then the type in question is not off_t.
>

Maybe you should try to transfer (using stock httpd) at least 3Gb file
on i386?

Also, I have the following in my archives:

---------- Forwarded message ----------
From: Marc Balmer <[hidden email]>
Date: Fri, Jun 20, 2008 at 11:09
Subject: Re: Apache doesn't handle files >2Gb properly
To: Alexey Suslikov <[hidden email]>
Cc: Marc Balmer <[hidden email]>, Jason McIntyre <[hidden email]>

* Alexey Suslikov wrote:
> Hello Marc, Jason.
>
> Can you comment on this issue?
>
> http://marc.info/?t=121379382200010&r=1&w=2
>
> Maybe it is a good idea to warn note about long files
> on 32-bit platform to httpd man page or/and FAQ?

no.  fix the problem, diffs are welcome ;)

maybe, if I find time, I look into this myself.

- Marc

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Otto Moerbeek
In reply to this post by Alexander Yurchenko-3
On Wed, Mar 11, 2009 at 06:48:49PM +0300, Alexander Yurchenko wrote:

> On Wed, Mar 11, 2009 at 05:37:14PM +0200, Alexey Suslikov wrote:
> > The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).
>
> huh?
>
> [grange@nohead tmp]$ cat x.c  
> #include <sys/types.h>
> #include <stdio.h>
>
> int main(void)
> {
>         printf("size of off_t is %u bits\n", 8 * sizeof(off_t));
> }
> [grange@nohead tmp]$ cc x.c  
> [grange@nohead tmp]$ ./a.out                                                  
> size of off_t is 64 bits
> [grange@nohead tmp]$ arch -s  
> i386

This code is not portable.  sizeof returns a size_t, which is 64 bits
on 64-bits platforms. Better use %zu.

        -Otto

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Ted Unangst-2
In reply to this post by Alexey Suslikov
On Wed, Mar 11, 2009 at 12:01 PM, Alexey Suslikov
<[hidden email]> wrote:
> On Wed, Mar 11, 2009 at 17:56, Ted Unangst <[hidden email]> wrote:
>> If using a 64-bit machine fixes it, then the type in question is not off_t.
>>
>
> Maybe you should try to transfer (using stock httpd) at least 3Gb file
> on i386?

I didn't say it wasn't a problem, I said your diagnosis was incorrect.

> Also, I have the following in my archives:
>>
>> http://marc.info/?t=121379382200010&r=1&w=2

from that very thread, the problem is using long *instead* of off_t.
Using off_t would fix the problem.

http://marc.info/?l=openbsd-bugs&m=121379609212212&w=2

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Alexey Suslikov
On Wed, Mar 11, 2009 at 19:11, Ted Unangst <[hidden email]> wrote:

> On Wed, Mar 11, 2009 at 12:01 PM, Alexey Suslikov
> <[hidden email]> wrote:
>> On Wed, Mar 11, 2009 at 17:56, Ted Unangst <[hidden email]> wrote:
>>> If using a 64-bit machine fixes it, then the type in question is not off_t.
>>>
>>
>> Maybe you should try to transfer (using stock httpd) at least 3Gb file
>> on i386?
>
> I didn't say it wasn't a problem, I said your diagnosis was incorrect.
>
>> Also, I have the following in my archives:
>>>
>>> http://marc.info/?t=121379382200010&r=1&w=2
>
> from that very thread, the problem is using long *instead* of off_t.
> Using off_t would fix the problem.
>
> http://marc.info/?l=openbsd-bugs&m=121379609212212&w=2
>

I can't recall exact code from httpd, but how about libexec/ftpd:

...
extern  off_t restart_point;
...
        if (restart_point) {
                if (type == TYPE_A) {
                        off_t i, n;
                        int c;

                        n = restart_point;
                        i = 0;
                        while (i++ < n) {
                                if ((c = getc(fin)) == EOF) {
                                        if (ferror(fin)) {
                                                perror_reply(550, name);
                                                goto done;
                                        } else
                                                break;
                                }
                                if (c == '\n')
                                        i++;
                        }
                } else if (lseek(fileno(fin), restart_point, SEEK_SET) < 0) {
                        perror_reply(550, name);
                        goto done;
                }
        }
...

Above code effectively results in inability to REST beyond 2Gb on
i386 however sequential read from position 0 will be ok.

Alexey

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Ted Unangst-2
On Wed, Mar 11, 2009 at 1:21 PM, Alexey Suslikov
<[hidden email]> wrote:
> I can't recall exact code from httpd, but how about libexec/ftpd:
>
> ...
> extern  off_t restart_point;
> ...

you're looking at the wrong code.

ftpcmd.y: restart_point = $4; /* XXX $4 is only "int" */

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

f.holop
In reply to this post by John Wong
hmm, on Wed, Mar 11, 2009 at 03:34:30AM +0800, John Wong said that
> I want to set up the web server to share file, but i know apache-1.3.x
> (which is openbsd default httpd) had the 4G file size limit, can i break
> this limit?

i dont remember such a limit, but i could be wrong.
but i definitely remember that thttpd had it, and lighttpd
did not.

-f
--
people have one thing in common, they're all different.

Reply | Threaded
Open this post in threaded view
|

Re: How to break the httpd's 4G file size limit?

Henning Brauer
In reply to this post by Alexey Suslikov
* Alexey Suslikov <[hidden email]> [2009-03-11 16:38]:
> The limitation is 2Gb on 32-bit platforms because of off_t (man lseek).

off_t is 64bit on all platforms we support. even vax.

--
Henning Brauer, [hidden email], [hidden email]
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam