KDC performance test - lookaside cache impact, testing framework

[Chris Hecker]

> KDC DB in OpenLDAP (same host as KDC):
> Without pre-authentication: 13 s, 14 s, 14s
> With pre-authentication: 6 s, 7 s, 6 s, 7 s

Are these times flipped?

Chris

On 2012/04/05 13:36, Petr Spacek wrote:
> Greetings,
> 
> my name is Petr Spacek and I work in cooperation with Red Hat on 
> Kerberos Performance Test Suite. (It's my master thesis project, some 
> details are at the end of e-mail, after the interesting part :-)
> 
> To get familiar with KDC and Kerberized environment I did some simple 
> synthetic performance tests on Fedora's MIT KDC version 1.10.1.
> I think results can be interesting for you.
> 
> Test was pretty simple: 100 kinits in parallel requested 100 TGTs each 
> (for single principal), i.e. totally 10000 TGT requests. Time necessary 
> to fulfil all requests was measured.
> These bursts were sent and measured one after another repeatedly.
> 
> KDC was on one computer, kinits was on another computer. Detailed HW 
> configurations are not important for now, I think. Load of KDC's CPU was 
> ~ 100 %, client's load was < 30 %.
> 
> Performance impact of some KDC configuration options was tested. Numbers 
> below are from configurations with disable_last_success = true and 
> disable_lockout = true. I can provide more details, if it will be necessary.
> 
> 
> The results are as follows (time measured for each successive burst):
> 
> KDC DB in local file:
> Without pre-authentication: 9 s, 24 s, 37 s, 48 s, 40 s, 40 s
> With pre-authentication: 26 s, 63 s, 75 s, 68 s, 72 s
> 
> KDC DB in OpenLDAP (same host as KDC):
> Without pre-authentication: 14 s, 36 s, 55 s, 55 s, 50 s
> With pre-authentication: 36 s, 86 s, 83 s
> 
> I was very surprised with there results. I repeated measurements and 
> variation between attempts was < 10 %. It's not very precise, but I 
> think it's enough to confirm observed trends.
> 
> 
> After some time a found "KDC lookaside cache" with defined "STALE_TIME" 
> 120 s. I think it explains observed trends: Time required to fulfil one 
> burst got stable after approximately 120 s. The reason is (I think) that 
> rate of adding new requests to the cache and removing "stale" requests 
> from the cache are approximately same. When size of the cache stabilizes 
> measured times also stabilizes. (It's only hypothesis.)
> 
> 
> Then I recompiled KDC with --disable-kdc-lookaside-cache switch and 
> repeated tests:
> 
> KDC DB in local file:
> Without pre-authentication: 6 s, 6 s, 6 s, 6 s
> With pre-authentication: 8 s, 8 s, 8 s, 8 s, 8 s
> 
> KDC DB in OpenLDAP (same host as KDC):
> Without pre-authentication: 13 s, 14 s, 14s
> With pre-authentication: 6 s, 7 s, 6 s, 7 s
> 
> 
> These numbers are nicer :-D
> 
> Please, let me ask some academic questions:
> Why the lookaside cache is there? I looked into RFC4120. Section 3.1.2 
> http://tools.ietf.org/html/rfc4120#section-3.1.2 say:
> 
>     Because Kerberos can run over unreliable transports such as UDP, the
>     KDC MUST be prepared to retransmit responses in case they are lost.
>     If a KDC receives a request identical to one it has recently
>     processed successfully, the KDC MUST respond with a KRB_AS_REP
>     message rather than a replay error.  In order to reduce ciphertext
>     given to a potential attacker, KDCs MAY send the same response
>     generated when the request was first handled.  KDCs MUST obey this
>     replay behavior even if the actual transport in use is reliable.
> 
> Ok, it's a standard and it has to be followed. Can be STALE_TIME lower? 
> It there a real problem, when lookaside cache is disabled? What are 
> implications?
> 
> 
> The used tests was really synthetic and far from real situations, I know 
> that. There was only point to point network, no firewalls, only TGT 
> requests and so on.
> 
> Now I work on general benchmarking framework. It should allow to 
> (relatively simply) write and run defined tests on bunch of machines in 
> parallel. It's aimed to end-to-end performance testing (test whole chain 
> from e.g. libpam_krb5 to KDC and back), but you can use plain kinit or 
> Kerberos library and create synthetic test. It's planed to be very general.
> 
> Would it be interesting for real Kerberos developers? I'm open to any 
> specific requests and proposals.
> 
> My work focuses on framework itself, but beside framework I will 
> implement real Kerberos tests. Some tests can be designed on you 
> requirements.
> 
> Thanks for your time.
>