[prads-devel] pcaps for testing and setup

[Gurvinder Singh]

Kacper Wysocki wrote:
> Edward Bjarte Fjellskål wrote:
>   
>> Edward Bjarte Fjellskål wrote:
>>     
>>> Edward Bjarte Fjellskål wrote:
>>>       
>>>> Hi List,
>>>> I also downloaded the Outside tcpdump data from:
>>>> 2000 DARPA Intrusion Detection Scenario Specific Data Sets
>>>> http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/2000data.html
>>>> http://www.ll.mit.edu/mission/communications/ist/corpora/ideval/data/2000/NT_dataset/outside.tcpdump.gz
>>>>         
>>> I ran the pcap 20 times, so this is more or less what is in there:
>>>       
>
> [snip]
>
>   
>>> -- Total assets detected                  :         479
>>> -- Total TCP service assets detected      :        1424
>>> -- Total TCP client assets detected       :         797
>>> -- Total UDP service assets detected      :           1
>>> -- Total UDP client assets detected       :           2
>>>       
>
>   
>> I did some refactoring, implemented some new features, and now
>> prads is running much much better. After one pass:
>>     
>
>   
>> -- Total assets detected                  :         707
>> -- Total TCP service assets detected      :        1374
>> -- Total TCP client assets detected       :         760
>> -- Total UDP service assets detected      :           2
>> -- Total UDP client assets detected       :           1
>>
>>
>> Night!
>>     
>
> [snip]
>
> Coolness! Here are some impactful changes :-)
> The goal is to get prads to snap up everything, of course, but that may
> not be possible when libpcap is working slower. On high-throughput links
> we will need either pcap_mmap functionality, PF_RING capturing or
> another gulp[*] methods.
>
> When I say "gulp" I mean we're trying to do the same - drinking from a
> fire hydrant.
>
> There are a number of things we can do in prads to make it scale -
> strategies include threading, stages, processes, async io, memory
> management, zero-copy, mmap, locking, zero-locking and more
> optimalization, but there are pros and cons to all of them.
>
> Specifically for cxtracker, and for scalability in general, recommended
> reading includes "The C10K problem" : http://www.kegel.com/c10k.html
> ... yep, hardware is not an issue anymore but oldschool programming
> doesn't cut it either.
>
> A real good approach to scalability is to take a look at what kind of
> system calls are worth avoiding due to the latency one can expect from
> them. Pretty graphs can be seen here: http://bulk.fefe.de/scalability/
> however one can make these graphs oneself, or think about when the
> computer isn't just executing code but also doing oh so expensive I/O.
>
> As I mentioned there are cons to these approaches as well - for example,
> with threading it's very easy to get into races and/or lock contention
> problems. A good survey can be found at
> http://pl.atyp.us/content/tech/servers.html
>
> 0K
>
> Gulp and other interesting links: http://staff.washington.edu/corey/gulp/
> _______________________________________________
> prads-devel mailing list
> prads-devel at projects.linpro.no
> http://projects.linpro.no/mailman/listinfo/prads-devel
>
>   
woowwww nice links and food for thoughts kacper...

Cheers,
Gurvinder