Subject: DRAFT status and Compatibility testing

From: John W Noerenberg II <[email-masked]>
Date: Tue, 28 Mar 2000 09:41:01 +0930

We've talked about moving 2440 to DRAFT for a while now.  We need to 
do this.  There are really two things we need: 1) two interoperable 
implementations, and 2) verification of the requirements of the 
protocol.  Even though we're not meeting in Adelaide, I thought 
kicking off some interoperability testing during the week would be an 
IETF'ish thing to do.

Towards 1, I propose the following:  We'll exchange encrypted, signed 
objects exercising the various modes described in 2440.  I'll keep 
score, and publish the results for the WG.  To participate, send me a 
note with subject:

	OpenPGP Compatibility Test 1

with directions on where to find a public key you'd like me to use. 
I will send an object encrypted with the keys I have and signed with 
mine on Thursday, Mar 30, 1500+930, at the conclusion of the SAAG 
meeting.

When you receive the message, decrypt the object, validate the 
signature, and send me back the results signed with your key.  Your 
response needs to include: the plaintext, what algorithms were used 
and evidence you've validated my signature.  My key is available on 
the MIT key server.

For 2, we'll need to build a compliance matrix to verify that all the 
required elements of 2440 have been treated (or not).  If we find 
features that have not been used, or are disputed, I will propose 
eliminating them unless resolution is desired, and found.  I'll start 
on 2, but I'll be looking for help.

Remember, the subject of the message is: OpenPGP Compatibility Test 
1.  I'm filtering messages by the subject to get your entry.  If you 
don't use this subject, I might miss you.

The message will go out at the end of the SAAG meeting on Thursday.

best,

-- 

john noerenberg
[email-masked]
   ----------------------------------------------------------------------
   But now I know our world is no more permanent
   than a wave rising on the ocean.
   -- Arthur Golden, "Memoirs of a Geisha", 1997
   ----------------------------------------------------------------------

---

From: "William H. Geiger III" <[email-masked]>
Date: Tue, 28 Mar 2000 10:03:59 -0600

In <a04310101b5059c6165cd@[169.208.192.29]>, on 03/27/00 

>score, and publish the results for the WG.  To participate, send me a 

IMHO what we need to do is put together a suite of test data that conforms
to RFC 2440.

Envelope
--------

Ascii-armor
PGP/MIME

Algorithms
----------

Compresion
Symetric encryption
Public Key encryption
Hash

Signatures
----------

Signature Types
Version 3
Version 4
    -- Hashed Subpackets
    -- Non-Hashed Subpackets

Keys
----

Private Keys
Public Keys
    Version 3
    Version 4
        -- Subkeys
    
(NOTE: This is not a complete list)

Additionally data sets should be divided into MUST, SHOULD, and OPTIONAL
catagories.

Testing should be done to see if the products are able to generate &
process the various packet types.

Some of the non-RFC2440 packets should be included (X.509, PhotoID,
...ect) so vendors can test that their software can properly handle
non-RFC2440 packets without crashing (this has been a problem with the pks
software).

IIRC someone had a partal data set available but I do not know if it is
still around nor if it has been updated.

-- 
---------------------------------------------------------------
William H. Geiger III                    http://www.openpgp.net  
Geiger Consulting    

Data Security & Cryptology Consulting
Programming, Networking, Analysis
 
PGP for OS/2:                   http://www.openpgp.net/pgp.html
---------------------------------------------------------------

---

From: Jon Callas <[email-masked]>
Date: Tue, 28 Mar 2000 10:51:09 -0800

In our last episode ("Re: DRAFT status and Compatibility testing", shown on

>IMHO what we need to do is put together a suite of test data that conforms

[...]

>non-RFC2440 packets without crashing (this has been a problem with the pks

I think this is an excellent idea. 2440 is a data format standard, and
consequently, the test suite mostly should consist of data. The only real
comment I have on your list is that PGP/MIME isn't part of 2440, it's a
separate RFC. (And in my opinion, this is a feature, not a bug. It's called
layering.)

There's a related issue that I want to bring up, though.

Many messages, when generated require random numbers to be used. In a
non-security environment, we'd have no objections to saying, "Here are the
inputs, here is the final data, make sure you generate this." Do we have an
objection to saying, "With a session key of K, encrypt message M, and your
result should look like R"? I don't -- I consider this to be the equivalent
of a crypto test vector. On the other hand, it still gives me a small
shudder, and I think this is one of those things on which gentlepersons can
disagree. An alternative is to come up with a test suite where an
implementation generates a message with known keys, and then writes out the
session keys and signature randoms, so that they can be verified. But on
this one, I don't see it's any better.

The problem, as I see it, is that if an implementation has a test mode in
which a random can be specified, then that's potentially a way to create a
hacked version with bad or known keys. But on the other hand, if an
implementation has a test mode in which the randoms can be leaked, then
that's a way to make a hacked version with a covert channel. If I have to
pick between one of those risks, I pick the former.

On the third hand, if we just say "non serviam" to that, then we run the
risk that a hacked version could pass all compliance tests and no one would
know otherwise. I don't like measuring compliance when we aren't specifying
the inputs, either.

I think my opinion (yes, I'm tentative here) is that 2440 is a data format
standard. We already have been wise enough to not get it into the whole
mess of crypto engineering in it. Our goal in advancing this is to make
sure that given a set of inputs, an implementation produces the proper
results. Those inputs may be finished messages that must be decoded, or
they may be plaintext, ciphers, randoms, etc. that should generate a fixed
message. There's an engineering problem in making an implementation
testable and secure. But that's a problem that the implementors have to
solve.

Opinions?

	Jon

---

From: "William H. Geiger III" <[email-masked]>
Date: Tue, 28 Mar 2000 14:06:36 -0600

In <p04310103b506a93ebb88@[172.20.1.38]>, on 03/28/00 
   at 12:51 PM, Jon Callas <[email-masked]> said:

>separate RFC. (And in my opinion, this is a feature, not a bug. It's
>called layering.)

Yes but there is no reason why not to include PGP/MIME test data. One can
document that this part of the data is for a seperate yet related RFC.
Most 2440 applications will also want to be PGP/MIME compliant.

>There's a related issue that I want to bring up, though.

Well I see the testing of the application as 2 part. The 1st part would be
the "inbound" processing of RFC 2440 packets. I don't see any risks
involved with using a fixed data set for this part of the testing.

The second part would be the "outbound" generation of RFC 2440 packets. We
should be able to have a set of "skeleton" packets that the tester could
use for compairison while ignoring the raw data (ie MPI's) that may be
different. As an example the application being tested needs to generate a
v3 RSA public key. The tests could use the "skeleton" test packet to
compair that all of the parts are generated and formated properly while
ignoring that data contained in MPI(1) and MPI(2) is different.

---

From: John W Noerenberg II <[email-masked]>
Date: Wed, 29 Mar 2000 09:34:46 +0930

>IMHO what we need to do is put together a suite of test data that conforms

Sounds like a good idea.  We'll need some data and specifications on 
what the data tests.  The message I proposed isn't an exhaustive 
test.  But at least it demonstrates some level of interoperability. 
That's a first step.  Are you volunteering to write the specs and 
build the test data?

>Additionally data sets should be divided into MUST, SHOULD, and OPTIONAL

First step here is to create the matrix I have in mind a la 1123.

>non-RFC2440 packets without crashing (this has been a problem with the pks

We can bear that in mind.  It certainly has impact on what we do.  If 
there are non-2440 packets we need to consider, I'd entertain the 
idea of an new draft to incorporate them into 2440's successor.
-- 

john noerenberg
[email-masked]

---

From: Will Price <[email-masked]>
Date: Tue, 28 Mar 2000 16:25:12 -0800

Some of the discussion I've seen thus far has gone a bit too far it
seems.

Proposals such as using pre-determined session keys or other
low-level crypto items may require ripping apart the software
packages being subjected to the testing and writing new code in order
to perform the test.  This is not going to prove anything other than
the fact that the new code (which isn't necessarily part of the
product) enabled a piece of software to complete an obscure low-level
test, and will lengthen the testing cycle from something that could
be done in a couple of weeks to something that will likely take many
months to agree upon, design, write, perform testing, repeat.

Any test designed to prove interoperability should adhere stricly to
the principle that interoperability between OpenPGP products is a
high-level operation basically limited to the following:

* Public/Private Key import/export
* Encrypted and/or signed message interop in its various forms

Things like PGP/MIME, and low-level testing of crypto algorithms are
irrelevant to this task except insofar as they have an effect on the
above two items (which for instance PGP/MIME does not -- that
document will require its own separate interop testing).  We're not
doing a FIPS crypto test on every product here, we're just testing
interop of the items generated by our common products which apply
directly to RFC 2440.

Making a test suite of messages and keys which use an array of
algorithms and variations should be a simple task any user can
perform sitting at any of our products without rewriting code,
turning on debug features, recompiling, etc...

This set of data would be termed the OpenPGP Compatibility Suite, and
each vendor could perform the testing on their own product, submit
that to the list, and any other vendor could use that same data to
verify the results.

- -- Will

Will Price, Director of Engineering
PGP Security, Inc.
a division of Network Associates, Inc.

---

From: Rodney Thayer <[email-masked]>
Date: Tue, 28 Mar 2000 21:23:26 -0800

I think that products should be built with "test points" in them, to allow
such things as interoperability testing.  This is how we've had to do
things in the IPsec and TLS worlds, and in other places.  I think we
should be able to "dump the keys", and I realize that means one
has to (gasp) have code that reveals the keys.

On the other hand, if we run in "trust me, it's ok" mode, we're degenerating
into security by obscurity.  If I were auditing an implementation, I'd want
to see the keys to validate it myself.

Remember, there's nothing stopping someone from revealing the keys
anyway -- you can't inhibit someone from saving copies of their
key material, or even publishing it.  So this is a previously
existing non-problem.

At 04:25 PM 3/28/00 -0800, Will Price wrote:
>-----BEGIN PGP SIGNED MESSAGE-----
>Hash: SHA1
>
>-----BEGIN PGP SIGNATURE-----
>Version: PGP 7.0 (Build 144 Alpha)
>
>iQA/AwUBOOFNMKy7FkvPc+xMEQJtdQCfTzZrTm9faAlZRfrgpSGfQjEbrqcAniGR
>9NHfkzlUee6nCdrK+Nr/DrIO
>=5URX
>-----END PGP SIGNATURE-----

---

From: Will Price <[email-masked]>
Date: Tue, 28 Mar 2000 22:00:37 -0800

Rodney Thayer wrote:
> 
> I think that products should be built with "test points" in them,
> to allow such things as interoperability testing.  This is how
> we've had to do things in the IPsec and TLS worlds, and in other
> places.  I think we should be able to "dump the keys", and I
> realize that means one

IPsec and TLS are hundreds of times more complicated to interop test
than OpenPGP should be.  IPsec and TLS are extremely complex, OpenPGP
is fairly straight forward.  Dumping keys is an obscure debug
scenario used to figure out why two implementations do not interop. 
Frankly, I expect the OpenPGP testing will have very few cases of
interop failure, and that such cases will be reasonably explainable
with things like "we didn't implement that optional algorithm."

> On the other hand, if we run in "trust me, it's ok" mode, we're
> degenerating into security by obscurity.  If I were auditing an
> implementation, I'd want to see the keys to validate it myself.

A main part of the point I'm making is that we are NOT auditing
implementations.  This is not a security review for each of our
products.  This is an interop test.  If some other product uses a
session key of 0, it's not the job of the interop tests to discover
that.  If I can read the other guy's messages/keys and he can read
mine, we're set to go.

> key material, or even publishing it.  So this is a previously

As I said, the tests must be able to be run by any user sitting at
each of our products without debug tools or special code.  Going
beyond that gets into a security review which is not what we're doing
here.

Hal Finney wrote:
> I am inclined to think that traditional interoperability testing is
> a better way to approach this problem.  At least with openpgp we
> don't all have to get together, it can all be done online.
> 
> We would define a specific set of message/key types to create, and
> everyone who wanted to participate would create messages/keys of
> those types (or whatever subset they support), then everyone else
> would try to read them.

Thanks for the correction Hal.  That sounds like an efficient
approach.

- -- Will

---

From: Werner Koch <[email-masked]>
Date: Wed, 29 Mar 2000 12:43:35 +0200

> Most 2440 applications will also want to be PGP/MIME compliant.

I don't think so, it is good Unix practise to do layering and MIME is
something you need in MUAs and not in an encryption tool.  

> use for compairison while ignoring the raw data (ie MPI's) that may be

It is not possible to simply compare the packets for a couple of reasons:
You should have random data in it, an implemenation has several 
choices to encode the packet length and and the order of subpackets is 
not regulated.  Doing some clever diffs over the output of a packet   
dumping tools would be possible.  

> v3 RSA public key. The tests could use the "skeleton" test packet to

(OpenPGP implementations SHOULD generate V4 keys (see 5.5.2))

IMHO, tTest scripts are the way to go; this is a lot of work unless we agree
on a common commandline interface and each implementation comes with a
wrapper for this interface.

   Werner

-- 
Werner Koch				OpenPGP key 621CC013	
OpenIT GmbH i.G.                        tel   +49 211 465357                           
Birkenstr. 12                           email [email-masked]                             
D-40233 Düsseldorf                      http://www.openit.de

---

From: Dave Del Torto <[email-masked]>
Date: Wed, 29 Mar 2000 03:19:54 -0800

>something you need in MUAs and not in an encryption tool.

Werner,

I am hoping you'll also implement our two newest drafts, OpenPGP/MIME
and OpenPGP/MIME Multipart Signatures. Both could be extremely useful
in a scriptable UNIX GnuPG version.

I've asked Will Price at PGP Security if they plan to support the
drafts, but I'm not sure they plan to do so. If both were
interoperable, that would of course be ideal. :)

    dave

---

From: Thomas Roessler <[email-masked]>
Date: Wed, 29 Mar 2000 13:23:54 +0200

On 2000-03-28 14:06:36 -0600, William H. Geiger III wrote:

> Yes but there is no reason why not to include PGP/MIME
> test data. One can document that this part of the data
> is for a seperate yet related RFC. Most 2440
> applications will also want to be PGP/MIME compliant.

I tend to disagree.  For instance, code signing with
OpenPGP certainly has no use for the PGP/MIME framework.
Additionally, PGP/MIME is essentially just about MIME
encapsulation of PGP messages, so a test suite for this
could be quite restricted, and would mostly have to handle
MIME encoding issues and pitfalls, plus some playing
around with micalg parameters and their effect.

-- 
http://www.guug.de/~roessler/

---

From: Thomas Roessler <[email-masked]>
Date: Wed, 29 Mar 2000 13:45:52 +0200

On 2000-03-29 03:19:54 -0800, Dave Del Torto wrote:

> I am hoping you'll also implement our two newest
> drafts, OpenPGP/MIME and OpenPGP/MIME Multipart
> Signatures. Both could be extremely useful in a
> scriptable UNIX GnuPG version.

I think that, for Unix use, a perl script based on, say,
the MIME::Tools (3pm) package would be better suited.
Everything else would mean code duplication when GnuPG is
used with any reasonable Mail User Agent which has a
decent MIME parser itself.

---

From: Werner Koch <[email-masked]>
Date: Wed, 29 Mar 2000 14:09:34 +0200

> in a scriptable UNIX GnuPG version.

I'll provide any low level function when needed; however everything
else can be done with a little bit of scripting.