cvs -z compression to reduce network traffic

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

cvs -z compression to reduce network traffic

Amit Kulkarni-5
Hi,

Are the various cvs mirrors allowing compression? I tried with cvs -z 5. I
currently sync from anoncvs3.usa and I think it doesn't, atleast the option
of tcpdump -A didn't show me any decompression activity, just ssh packets
being sent. top also didn't show any unzip or tar in the -I option....

If any mirror admin allows compression, please let me/us know. If they are
willing to publicize the allowed compression level, please put in the list
of cvs mirrors page!

Syncing to src, ports, xenocara wastes many MB per month per person...and
any help would be appreciated to cut down network traffic. I would be
willing to be test this if it is not enabled currently, and a cvs server
admin would like to enable it and check the load.

thanks in advance

Reply | Threaded
Open this post in threaded view
|

Re: cvs -z compression to reduce network traffic

Nick Holland
On 07/16/13 22:25, Amit Kulkarni wrote:

> Hi,
>
> Are the various cvs mirrors allowing compression? I tried with cvs -z 5. I
> currently sync from anoncvs3.usa and I think it doesn't, atleast the option
> of tcpdump -A didn't show me any decompression activity, just ssh packets
> being sent. top also didn't show any unzip or tar in the -I option....
>
> If any mirror admin allows compression, please let me/us know. If they are
> willing to publicize the allowed compression level, please put in the list
> of cvs mirrors page!
>
> Syncing to src, ports, xenocara wastes many MB per month per person...and
> any help would be appreciated to cut down network traffic. I would be
> willing to be test this if it is not enabled currently, and a cvs server
> admin would like to enable it and check the load.
>
> thanks in advance
>

how about doing your compression at the SSH transport level, rather than
the cvs level?

something like a .ssh/config:
host MyFavMirror.com
   Compression yes

Be forewarned, I've seen /some/ systems do a horrible slow-down with
compression, but since your concern is bandwidth, probably not your issue.

Nick.

Reply | Threaded
Open this post in threaded view
|

Re: cvs -z compression to reduce network traffic

Stuart Henderson
In reply to this post by Amit Kulkarni-5
On 2013-07-17, Amit Kulkarni <[hidden email]> wrote:
> Hi,
>
> Are the various cvs mirrors allowing compression? I tried with cvs -z 5. I
> currently sync from anoncvs3.usa and I think it doesn't, atleast the option
> of tcpdump -A didn't show me any decompression activity, just ssh packets
> being sent. top also didn't show any unzip or tar in the -I option....

Why would it show up as tar or unzip? It isn't sending files around, it's
a client-server network protocol.

tcpdump -A won't show you anything useful for anoncvs-over-ssh because it's
inside an SSH session, you'll just see the encrypted packets. You would
notice if you were using pserver (if there are any servers left still
offering it). I don't think server admins would go out of their way to
disable it though..

> Syncing to src, ports, xenocara wastes many MB per month per person...and
> any help would be appreciated to cut down network traffic. I would be
> willing to be test this if it is not enabled currently, and a cvs server
> admin would like to enable it and check the load.

cvsync is generally more suitable for frequent syncing. If there are no
changes to the repository, it typically sends about 8MB from the client to
the server, and about 100KB from the server to the client. I haven't
measured bandwidth for cvs-over-ssh but I bet it's more, especially if
you have local diffs.

Compression is likely to be available with all cvsync servers (default
is level 1, mine uses level 5), though it doesn't really help with the
upstream bandwidth use, only really makes a difference if there are big
changes to the repository.

Reply | Threaded
Open this post in threaded view
|

Re: cvs -z compression to reduce network traffic

Jan Stary
In reply to this post by Amit Kulkarni-5
On Jul 16 21:25:34, [hidden email] wrote:
> Hi,
>
> Are the various cvs mirrors allowing compression? I tried with cvs -z 5. I
> currently sync from anoncvs3.usa and I think it doesn't, atleast the option
> of tcpdump -A didn't show me any decompression activity, just ssh packets
> being sent. top also didn't show any unzip or tar in the -I option....

The compression and decompression happens within the client/server
- there is no tar or unzip or whatever extra process involved.
Also, how would tcpdump "show decompression activity"?
Of course it just shows the ssh packets.

> If any mirror admin allows compression, please let me/us know. If they are
> willing to publicize the allowed compression level, please put in the list
> of cvs mirrors page!

I believe the mirrors allow whatever compression you ask for.

> Syncing to src, ports, xenocara wastes many MB per month per person...and
> any help would be appreciated to cut down network traffic.

Per person?  Why don't you just sync to one place,
and let the local users sync from that? That alone
will cut your traffic by a factor of 1/persons.

Also, how "many MB per month" is it actually,
and what percentage of your monthly traffic is it?

I wouldn't be surprised that

(1) you are actually using the cvs -z compression already
(2) the savings of compressing some more are negligible
    in your overal monthly traffic
(3) using ssh compression makes the inner cvs compression irrelevant
    (and someone has numbers to back that up)


        Jan

Reply | Threaded
Open this post in threaded view
|

Re: cvs -z compression to reduce network traffic

Juan Francisco Cantero Hurtado
In reply to this post by Amit Kulkarni-5
On Tue, Jul 16, 2013 at 09:25:34PM -0500, Amit Kulkarni wrote:

> Hi,
>
> Are the various cvs mirrors allowing compression? I tried with cvs -z 5. I
> currently sync from anoncvs3.usa and I think it doesn't, atleast the option
> of tcpdump -A didn't show me any decompression activity, just ssh packets
> being sent. top also didn't show any unzip or tar in the -I option....
>
> If any mirror admin allows compression, please let me/us know. If they are
> willing to publicize the allowed compression level, please put in the list
> of cvs mirrors page!
>
> Syncing to src, ports, xenocara wastes many MB per month per person...and
> any help would be appreciated to cut down network traffic. I would be
> willing to be test this if it is not enabled currently, and a cvs server
> admin would like to enable it and check the load.
>
> thanks in advance

I use stuart's and nick's tricks almost daily. cvsync for everything
except when I break something in some port, in this case I just update
the directory from the cvs.

Probably other good option in your situation is to use a git or
mercurial mirror. Both are very efficients in the network use. The
problem is the public git mirrors are managed by people external to the
project (and sometimes converted with broken tools).

I did some tests some weeks ago with "git-cvs" and "hg convert" but IIRC
both use a lot of RAM. My intention was to run a public mirror of ports,
src and xenocara but I can't ask for a server with so many RAM to some
sponsor for a mostly useless project.

--
Juan Francisco Cantero Hurtado http://juanfra.info

Reply | Threaded
Open this post in threaded view
|

Re: cvs -z compression to reduce network traffic

Amit Kulkarni-5
On Thu, 18 Jul 2013 03:06:03 +0200
Juan Francisco Cantero Hurtado <[hidden email]> wrote:

> On Tue, Jul 16, 2013 at 09:25:34PM -0500, Amit Kulkarni wrote:
> > Hi,
> >
> > Are the various cvs mirrors allowing compression? I tried with cvs -z 5. I
> > currently sync from anoncvs3.usa and I think it doesn't, atleast the option
> > of tcpdump -A didn't show me any decompression activity, just ssh packets
> > being sent. top also didn't show any unzip or tar in the -I option....
> >
> > If any mirror admin allows compression, please let me/us know. If they are
> > willing to publicize the allowed compression level, please put in the list
> > of cvs mirrors page!
> >
> > Syncing to src, ports, xenocara wastes many MB per month per person...and
> > any help would be appreciated to cut down network traffic. I would be
> > willing to be test this if it is not enabled currently, and a cvs server
> > admin would like to enable it and check the load.
> >
> > thanks in advance
>
> I use stuart's and nick's tricks almost daily. cvsync for everything
> except when I break something in some port, in this case I just update
> the directory from the cvs.
>
> Probably other good option in your situation is to use a git or
> mercurial mirror. Both are very efficients in the network use. The
> problem is the public git mirrors are managed by people external to the
> project (and sometimes converted with broken tools).
>
> I did some tests some weeks ago with "git-cvs" and "hg convert" but IIRC
> both use a lot of RAM. My intention was to run a public mirror of ports,
> src and xenocara but I can't ask for a server with so many RAM to some
> sponsor for a mostly useless project.
>

Thanks folks, for all of your kind suggestions and cluestick from Nick and Stuart. I will convert to cvsync.