[prads-devel] pcaps for testing and setup

[Kacper Wysocki]

Edward Bjarte Fjellskål wrote:
> Edward Bjarte Fjellskål wrote:
>> Edward Bjarte Fjellskål wrote:
>>> Hi List,
>>> I also downloaded the Outside tcpdump data from:
>>> 2000 DARPA Intrusion Detection Scenario Specific Data Sets
>>> http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/2000data.html
>>> http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/2000/NT_dataset/outside.tcpdump.gz
>> I ran the pcap 20 times, so this is more or less what is in there:

[snip]

>> -- Total assets detected                  :         479
>> -- Total TCP service assets detected      :        1424
>> -- Total TCP client assets detected       :         797
>> -- Total UDP service assets detected      :           1
>> -- Total UDP client assets detected       :           2

> I did some refactoring, implemented some new features, and now
> prads is running much much better. After one pass:

> -- Total assets detected                  :         707
> -- Total TCP service assets detected      :        1374
> -- Total TCP client assets detected       :         760
> -- Total UDP service assets detected      :           2
> -- Total UDP client assets detected       :           1
> 
> 
> Night!

[snip]

Coolness! Here are some impactful changes :-)
The goal is to get prads to snap up everything, of course, but that may
not be possible when libpcap is working slower. On high-throughput links
we will need either pcap_mmap functionality, PF_RING capturing or
another gulp[*] methods.

When I say "gulp" I mean we're trying to do the same - drinking from a
fire hydrant.

There are a number of things we can do in prads to make it scale -
strategies include threading, stages, processes, async io, memory
management, zero-copy, mmap, locking, zero-locking and more
optimalization, but there are pros and cons to all of them.

Specifically for cxtracker, and for scalability in general, recommended
reading includes "The C10K problem" : http://www.kegel.com/c10k.html
... yep, hardware is not an issue anymore but oldschool programming
doesn't cut it either.

A real good approach to scalability is to take a look at what kind of
system calls are worth avoiding due to the latency one can expect from
them. Pretty graphs can be seen here: http://bulk.fefe.de/scalability/
however one can make these graphs oneself, or think about when the
computer isn't just executing code but also doing oh so expensive I/O.

As I mentioned there are cons to these approaches as well - for example,
with threading it's very easy to get into races and/or lock contention
problems. A good survey can be found at
http://pl.atyp.us/content/tech/servers.html

0K

Gulp and other interesting links: http://staff.washington.edu/corey/gulp/