httpdump?

classic Classic list List threaded Threaded
15 messages Options
Reply | Threaded
Open this post in threaded view
|

httpdump?

Jeff Simmons
Anyone know of a text-based program that will dump http protocol packets?
Like tcpdump, but for http.

--
Jeff Simmons                                   [hidden email]
Simmons Consulting - Network Engineering, Administration, Security
"You guys, I don't hear any noise.  Are you sure you're doing it right?"
        --  My Life With The Thrill Kill Kult

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Pui Edylie
why not tcpdump and filter it on port 80?

Jeff Simmons wrote:
> Anyone know of a text-based program that will dump http protocol packets?
> Like tcpdump, but for http.

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Jeff Simmons
I need, at a minimum, which virtual server at a particular IP address is being
accessed, and the contents of any GET commands (methods). If there's a way to
get this via tcpdump I haven't found it yet.

On Wednesday 19 November 2008 19:52, Pui Edylie wrote:
> why not tcpdump and filter it on port 80?
>
> Jeff Simmons wrote:
> > Anyone know of a text-based program that will dump http protocol packets?
> > Like tcpdump, but for http.

--
Jeff Simmons                                   [hidden email]
Simmons Consulting - Network Engineering, Administration, Security
"You guys, I don't hear any noise.  Are you sure you're doing it right?"
        --  My Life With The Thrill Kill Kult

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

ropers
In reply to this post by Jeff Simmons
2008/11/20 Jeff Simmons <[hidden email]>:
> Anyone know of a text-based program that will dump http protocol packets?
> Like tcpdump, but for http.

I'm not an expert, but I here are my dimly and pseudo-educated guesses
and hunches and 2 * n  eurocents:

- In order to selectively *only* capture HTTP packets, you have to
*recognize* which TCP packets are/contain HTTP packets first.

- Granted, a HTTP packet may not arrive via TCP, since according to Wikipedia:
> HTTP is not constrained to using TCP/IP and its supporting layers, although this is its most popular application on the Internet. Indeed HTTP can be "implemented on top of any other protocol on the Internet, or on other networks.
HOWEVER, even when using another kind of networking stack, you still
have to figure out which packets are HTTP packets and which aren't.
Also, TCP/IP is ubiquitous, and
J-Random-Hacker's-Own-Leet-Networking-Protocol isn't.

- It is my understanding that with TCP/IP at least, pretty much the
only way to determine whether what you've got at your hands is an HTTP
packet is to actually look at it.

- Thus, identifying all HTTP packets requires capturing each TCP
packet, or at least capture part of each TCP packet.

- Even if we were to assume that it was possible to only capture part
of each packet and then determine --in real time, while you're dumping
TCP packets-- what the HTTP packets are and then only fully capturing
those, this would probably be needlessly complex and waste more
resources than not capturing all of the packets in full saves.

- For the above reasons, you're probably much better off to just
capture all of your TCP packets on a given interface with tcpdump, and
then selectively reassemble only what you want with tcpflow. Also for
the above reasons, I am not convinced a dedicated "httpdump" tool
would be particularly useful.

Again, I don't really know what I'm talking about to a great extent in
this area, but that's AFAIK.

'hope this helps,
--ropers

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

ropers
Sorry about the grammar mistakes. Fixed below.

2008/11/20 ropers <[hidden email]>:

> 2008/11/20 Jeff Simmons <[hidden email]>:
>> Anyone know of a text-based program that will dump http protocol packets?
>> Like tcpdump, but for http.
>
> I'm not an expert, but I here are my dimly and pseudo-educated guesses
> and hunches and 2 * n  eurocents:
>
> - In order to selectively *only* capture HTTP packets, you have to
> *recognize* which TCP packets are/contain HTTP packets first.
>
> - Granted, a HTTP packet may not arrive via TCP, since according to Wikipedia:
>> HTTP is not constrained to using TCP/IP and its supporting layers, although this is its most popular application on the Internet. Indeed HTTP can be "implemented on top of any other protocol on the Internet, or on other networks.
> HOWEVER, even when using another kind of networking stack, you still
> have to figure out which packets are HTTP packets and which aren't.
> Also, TCP/IP is ubiquitous, and
> J-Random-Hacker's-Own-Leet-Networking-Protocol isn't.
>
> - It is my understanding that with TCP/IP at least, pretty much the
> only way to determine whether what you've got at your hands is an HTTP
> packet is to actually look at it.
>
> - Thus, identifying all HTTP packets requires capturing each TCP
> packet, or at least capturing part of each TCP packet.
>
> - Even if we were to assume that it was possible to only capture part
> of each packet and then determine --in real time, while you're dumping
> TCP packets-- what the HTTP packets are and then only fully capture
> those, this would probably be needlessly complex and waste more
> resources than not capturing all of the packets in full might save.
>
> - For the above reasons, you're probably much better off just
> capturing all of your TCP packets on a given interface with tcpdump, and
> then selectively reassembling only what you want with tcpflow. Also for
> the above reasons, I am not convinced a dedicated "httpdump" tool
> would be particularly useful.
>
> Again, I don't really know what I'm talking about to a great extent in
> this area, but that's AFAIK.
>
> 'hope this helps,
> --ropers

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

John
In reply to this post by Jeff Simmons
On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:

> I need, at a minimum, which virtual server at a particular IP address is being
> accessed, and the contents of any GET commands (methods). If there's a way to
> get this via tcpdump I haven't found it yet.
>
> On Wednesday 19 November 2008 19:52, Pui Edylie wrote:
> > why not tcpdump and filter it on port 80?
> >
> > Jeff Simmons wrote:
> > > Anyone know of a text-based program that will dump http protocol packets?
> > > Like tcpdump, but for http.

Try netwox, tethereal, tcpflow.  One of those should get you what you
want.  Not necessarily in that order though.

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

John Wright-6
In reply to this post by Jeff Simmons
On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:
> I need, at a minimum, which virtual server at a particular IP address is being
> accessed, and the contents of any GET commands (methods). If there's a way to
> get this via tcpdump I haven't found it yet.

Just increase the snaplen.

tcpdump -s 65000 -w dump port 80

^C when you're done and vim the dump.  Raw packets but you'll see the GET in
there.

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Stuart Henderson
In reply to this post by John
On 2008-11-20, John Jackson <[hidden email]> wrote:
> On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:
>> I need, at a minimum, which virtual server at a particular IP address is being
>> accessed, and the contents of any GET commands (methods). If there's a way to
>> get this via tcpdump I haven't found it yet.

urlsnarf from dsniff
tcpdump -X -s1500
ngrep

> ...tethereal...

please, just rm that now...

if it's still called "tethereal" it's super-old and has loads of
known security problems.

if you're even considering that risky software, you should at
least be running the latest version (1.0.4), capture the files
offline with tcpdump -w, and manually run tshark on the capture
file as an unprivileged user. (this is NOT a recommendation to
use wireshark, but if you're going to do it anyway, be as safe
as you can about it).

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Jeff Simmons
In reply to this post by John
On Wednesday 19 November 2008 20:48, John Jackson wrote:

> On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:
> > I need, at a minimum, which virtual server at a particular IP address is
> > being accessed, and the contents of any GET commands (methods). If
> > there's a way to get this via tcpdump I haven't found it yet.
> >
> > On Wednesday 19 November 2008 19:52, Pui Edylie wrote:
> > > why not tcpdump and filter it on port 80?
> > >
> > > Jeff Simmons wrote:
> > > > Anyone know of a text-based program that will dump http protocol
> > > > packets? Like tcpdump, but for http.
>
> Try netwox, tethereal, tcpflow.  One of those should get you what you
> want.  Not necessarily in that order though.

tcpflow (which is in ports) is giving me exactly what I need, with a little
help from perl. I obviously need to improve my "search for a program that
does X" skills.

Thanks to everyone for the help.

--
Jeff Simmons                                   [hidden email]
Simmons Consulting - Network Engineering, Administration, Security
"You guys, I don't hear any noise.  Are you sure you're doing it right?"
        --  My Life With The Thrill Kill Kult

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

amarendra godbole
In reply to this post by Jeff Simmons
On Thu, Nov 20, 2008 at 9:48 AM, Jeff Simmons <[hidden email]> wrote:

> I need, at a minimum, which virtual server at a particular IP address is being
> accessed, and the contents of any GET commands (methods). If there's a way to
> get this via tcpdump I haven't found it yet.
>
> On Wednesday 19 November 2008 19:52, Pui Edylie wrote:
>> why not tcpdump and filter it on port 80?
>>
>> Jeff Simmons wrote:
>> > Anyone know of a text-based program that will dump http protocol packets?
>> > Like tcpdump, but for http.
[...]

tshark, the text-base capture tool of wireshark (ethereal) should get
you what you want. You may have to setup filters though.

-Amarendra

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Jeff Simmons
On Saturday 22 November 2008 18:19, you wrote:
> On Thu, Nov 20, 2008 at 9:48 AM, Jeff Simmons <[hidden email]>
wrote:

> > I need, at a minimum, which virtual server at a particular IP address is
> > being accessed, and the contents of any GET commands (methods). If
> > there's a way to get this via tcpdump I haven't found it yet.
> >
> > On Wednesday 19 November 2008 19:52, Pui Edylie wrote:
> >> why not tcpdump and filter it on port 80?
> >>
> >> Jeff Simmons wrote:
> >> > Anyone know of a text-based program that will dump http protocol
> >> > packets? Like tcpdump, but for http.
>
> [...]
>
> tshark, the text-base capture tool of wireshark (ethereal) should get
> you what you want. You may have to setup filters though.

According to their web site, Wireshark needs gcc 4.x to compile. It also needs
to be run behind an airwall.

--
Jeff Simmons                                   [hidden email]
Simmons Consulting - Network Engineering, Administration, Security
"You guys, I don't hear any noise.  Are you sure you're doing it right?"
        --  My Life With The Thrill Kill Kult

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Stuart Henderson
On 2008-11-23, Jeff Simmons <[hidden email]> wrote:
>>
>> tshark, the text-base capture tool of wireshark (ethereal) should get
>> you what you want. You may have to setup filters though.

> According to their web site, Wireshark needs gcc 4.x to compile.

then their web site is wrong.

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Jacek Artymiak (devGuide.net)
In reply to this post by John Wright-6
On Thu, Nov 20, 2008 at 11:01 AM, John Wright <[hidden email]> wrote:

> On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:
>> I need, at a minimum, which virtual server at a particular IP address is being
>> accessed, and the contents of any GET commands (methods). If there's a way to
>> get this via tcpdump I haven't found it yet.
>
> Just increase the snaplen.
>
> tcpdump -s 65000 -w dump port 80
>
> ^C when you're done and vim the dump.  Raw packets but you'll see the GET in
> there.

You can make the dump ASCII-safe with strings(1).

--
Jacek Artymiak
http://devGuide.net

Installing OpenBSD 4.4 web seminar
http://www.devguide.net/training/webinars/openbsd001

vi(1) Tips: Essential vi/vim Editor Skills, 1st ed.
http://www.devguide.net/books/vitips1

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

ropers
>> On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:

>> Just increase the snaplen.
>>
>> tcpdump -s 65000 -w dump port 80

With some tcpdump(8) versions on non-OpenBSD Unix-like OSes (e.g.
tcpdump version 3.9.8/Ubuntu 8.10), the man page says:

 -s     Snarf snaplen bytes of data from each packet (...) Set‐
              ting snaplen to 0 means use the required length to catch
whole packets.

The man page for OpenBSD's tcpdump doesn't mention anything about
setting the snaplen to 0, and trying to invoke OpenBSD's tcpdump with
-s 0 results in an error of:

> tcpdump: invalid snaplen 0

(tested with OpenBSD 4.3 GENERIC)

This is probably a naive question, but how would one best replicate
the -s 0 functionality with OpenBSD's tcpdump? Is there a reason why
Jeff specifically suggested -s 65000?

Many thanks and regards,
--ropers

Reply | Threaded
Open this post in threaded view
|

Re: httpdump?

Pierre Riteau-3
2008/11/23 ropers <[hidden email]>:

>>> On Wed, Nov 19, 2008 at 08:18:00PM -0800, Jeff Simmons wrote:
>
>>> Just increase the snaplen.
>>>
>>> tcpdump -s 65000 -w dump port 80
>
> With some tcpdump(8) versions on non-OpenBSD Unix-like OSes (e.g.
> tcpdump version 3.9.8/Ubuntu 8.10), the man page says:
>
>  -s     Snarf snaplen bytes of data from each packet (...) Set‐
>              ting snaplen to 0 means use the required length to catch
> whole packets.
>
> The man page for OpenBSD's tcpdump doesn't mention anything about
> setting the snaplen to 0, and trying to invoke OpenBSD's tcpdump with
> -s 0 results in an error of:
>
>> tcpdump: invalid snaplen 0
>
> (tested with OpenBSD 4.3 GENERIC)
>
> This is probably a naive question, but how would one best replicate
> the -s 0 functionality with OpenBSD's tcpdump? Is there a reason why
> Jeff specifically suggested -s 65000?
>
> Many thanks and regards,
> --ropers
>
>

Since the size of a IPv4 packet is coded on 16 bits it can't be more
than 65536 octets.
So -s 65536 and -s 0 should behave the same for IPv4 packets.
Note that IPv6 has a jumbogram feature enabling the use of bigger
packets but I never saw it in action.

And since tcpdump doesn't reassemble framented IP packets, your
network interface MTU should be enough anyway.

--
Pierre Riteau