KDC performance test - lookaside cache impact, testing framework

[Petr Spacek]

On 05/24/2012 08:11 AM, Greg Hudson wrote:
> On 05/17/2012 07:59 AM, Petr Spacek wrote:
>> Another question: Are there any public patches with proposals for better
>> lookaside cache? I'm curious ... :-)
>
> If you're still curious, I just put together this:
>
>      https://github.com/greghudson/krb5/tree/lookaside
>
> which uses a constant-size hash table defaulting to 16384 buckets; this
> can be modified with the LOOKASIDE_HASH_SIZE preprocessor variable.  The
> code would probably need a bit more work before going into the master
> (mostly, the overly-clever linked list implementations need to be better
> commented).
>
> We could also consider making the hash table configurably or dynamically
> sized, but I don't want to go overboard.

Sorry for a big delay.

I applied your changes from commits 14154d05bccee9dc673894568960a0a3b0274460 
and d22bc2e599918b8adfa38bf554dbc8171082337e to source used in previous tests.

The results from repeated tests (= AS_REQ flood, 180 seconds) are impressive:
KDC performance with hash-based lookaside cache is nearly same as without 
lookaside cache. You can compare performances on the following graph:
http://people.redhat.com/~pspacek/a/2012-06-15/as_flood_test.pdf


The fast KDC has one disadvantage: Lookaside cache can consume huge amount of 
memory.

I captured output from ps -F before and after each test. Data were captured 
few seconds after the test end. At the time of capture no unserviced requests 
should exist. Swap usage was few hundreds of kB. Approximately 3 MB/s of 
traffic (in each direction) flown through the network interface during the 
test (in a fastest case).

ps output about KDC processes (command was cut down to shorten lines):

KDC without lookaside cache - before the test:
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root      2191     1  0 38187  1028   0 13:18 ?        00:00:00 krb5kdc
- after the test:
root      2191     1 83 38187  1520   0 13:18 ?        00:02:37 krb5kdc


KDC with original lookaside cache - before the test:
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root      3589     1  0 38187  1024   0 16:35 ?        00:00:00 krb5kdc
- after the test:
root      3589     1 94 47862 40456   0 16:35 ?        00:05:03 krb5kdc


KDC with hash-based lookaside cache - before the test:
UID        PID  PPID  C    SZ   RSS PSR STIME TTY          TIME CMD
root      1924     1  0 38219  1148   0 13:03 ?        00:00:00 krb5kdc
- after the test:
root      1924     1 88 185974 592900 0 13:03 ?        00:02:49 krb5kdc


KDC with the new cache is able to handle much bigger amount of requests in 
same time. In this case KDC consumes around 600 MB of memory because each 
request is stored the lookaside cache. (If I understood it correctly...)

This test is really extreme, I know, but some memory usage limit could be good 
as anti-DoS measure.


Let me know if you are interested in some other tests or additional 
information is needed.

Best regards,
Petr Spacek