Received: by ns.secondary.com (8.9.3/8.9.3) id XAA13415 for ietf-pop3ext-bks; Mon, 31 Jan 2000 23:58:49 -0800 (PST) Received: from demo.esys.ca (demo.esys.ca [207.167.22.130]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id XAA13411 for ; Mon, 31 Jan 2000 23:58:48 -0800 (PST) Received: from messagingdirect.com (2-028-edm.dial.worldgate.ca [207.167.2.28]) by demo.esys.ca (2.0.4/SMS 2.0.4-beta-5) with ESMTP id BAA00729; Tue, 1 Feb 2000 01:00:58 -0700 Message-ID: <389691B7.98B12E09@messagingdirect.com> Date: Tue, 01 Feb 2000 00:56:39 -0700 From: Alexey Melnikov X-Mailer: Mozilla 4.7 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Solar Designer CC: Anders Rundegren , ietf-pop3ext@imc.org Subject: Re: Extended RETR - Draft posted References: <200001310419.HAA05533@false.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: Solar Designer wrote: > Yes, it seems we need that. There's one more issue that you'd need > to decide on -- are we calculating CRLF's the way they're transferred > over POP3 (would be more essential for the protocol), or the way line > separators are typically stored on a UNIX-based server (to allow for > a simple seek). There are servers that 1). construct message on the flight, storing it in some different format. 2). store headers separately from body. 3). store message with CRLF even if they run on Unix 4). store message in bytestuffed form (as received from SMTP) Whatever rule you choose for calculation of offset you will not satisfy ALL implementations, so there is no point doing that. BTW, ESMTP Checkpoint/Restart extension (RFC 1845) that does exactly the same thing for SMTP (and suffers from the same dot stuffing problem) says: Any octets added by any SMTP data-stuffing algorithm do not count as part of this offset. In the case of data transferred with the DATA command the offset must also correspond to the beginning of a line. In checkpoint/restart offset starts from 0. I would suggest to use the same for consistency. > > The model should be very simple, please correct me if I'm wrong: > > > > msg-offset-bytestuff-server-client-bytedestuff-msg > > Unfortunately, there may be a few more conversions: > > server_mailbox -> skip(*)_to_offset -> bytestuff -> LF_to_CRLF -> ... > ... transfer -> bytedestuff -> convert_to_client_mailbox_format > > (*) we haven't defined the "skip" for the server_mailbox "layer", yet. > > > Example: > > > > Message number is '1'. > > Message size is 217,743 bytes. > > Offset 113,435 starts with a '.'. > > > > Lets assume our transfer got interrupted after 113,434 bytes. Now, the > > To illustrate the case of having to byte-stuff, you'd want the > transfer to be interrupted after 113,435 bytes. Would it be easier to use TOP-like command? Something like TOP Alexey Received: by ns.secondary.com (8.9.3/8.9.3) id XAA12691 for ietf-pop3ext-bks; Mon, 31 Jan 2000 23:35:54 -0800 (PST) Received: from demo.esys.ca (demo.esys.ca [207.167.22.130]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id XAA12687 for ; Mon, 31 Jan 2000 23:35:52 -0800 (PST) Received: from messagingdirect.com (2-028-edm.dial.worldgate.ca [207.167.2.28]) by demo.esys.ca (2.0.4/SMS 2.0.4-beta-5) with ESMTP id AAA00706; Tue, 1 Feb 2000 00:37:54 -0700 Message-ID: <38967A3D.948E6851@messagingdirect.com> Date: Mon, 31 Jan 2000 23:16:29 -0700 From: Alexey Melnikov X-Mailer: Mozilla 4.7 [en] (WinNT; U) X-Accept-Language: en MIME-Version: 1.0 To: Anders Rundegren , POP3 Extensions Mailing List Subject: Re: Extended RETR - Draft posted References: <182081.3157782802@nifty-jr.innosoft.com> <3892DE6A.CD19311@rundegren.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: Anders Rundegren wrote: > > The draft for the extended RETR command has now been posted. Feedback > is, as always, very appreciated. I'm not sure about the section dealing > with extended response codes, it definitely needs some more work. Since > English is not my native language I ask you to look over this too. > > http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt Few comments: 1). 1. Introduction: >Many clients connecting to an ISP, or even ISP itself, >***have their modems improperly setup*** ... I am not sure why you want to blame modem configuration for all problems. I use to have 19200 dialup connection to my ISP and a very noisy phone line. Sometimes connection was dropped every 5 minutes. 2). 2.2 The offset argument If the specified offset is beyond the size of the message being retrieved, or in any other way erroneous in its syntax, ***the response from the server should start with "-ERR".**** I suggest to change the ending to something like the following: the server should send negative response code. 3). 3. Extended Response Codes Change -ERR message exists [OFFSET-OVERRUN] and -ERR no such message [NON-EXISTENT] to -ERR [OFFSET-OVERRUN] message exists and -ERR [NON-EXISTENT] no such message respectively This is according to ABNF in RFC 2449. I also doubt the usefulness of [NON-EXISTENT] response. Good behaving client must check the validity of the message number when it reconnects. Also if another client deleted the interrupted message while the client reconnects, the client is risking to resume downloading the next message. This is not what you expect from the client. In this case some persistent message Id should be used (UIDL?). Alexey Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id UAA29522 for ietf-pop3ext-bks; Sun, 30 Jan 2000 20:18:54 -0800 (PST) Received: from mystic.false.com (root@false.com [198.65.171.171]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id UAA29515 for ; Sun, 30 Jan 2000 20:18:53 -0800 (PST) Received: from false.com (root@false.com [198.65.171.171]) by mystic.false.com (8.9.3/8.9.3) with ESMTP id WAA23913; Sun, 30 Jan 2000 22:20:57 -0600 Received: (from solar@localhost) by false.com (8.8.5/8.8.5) id HAA05533; Mon, 31 Jan 2000 07:19:30 +0300 From: Solar Designer Message-Id: <200001310419.HAA05533@false.com> Subject: Re: Extended RETR - Draft posted In-Reply-To: <3894FEF4.1D61AFF7@rundegren.com> from Anders Rundegren at "Jan 31, 0 04:18:12 am" To: anders@rundegren.com (Anders Rundegren) Date: Mon, 31 Jan 2000 07:19:29 +0300 (MSK) Cc: ietf-pop3ext@imc.org X-Mailer: ELM [version 2.4ME+ PL31 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: > > "The offset is given in octets. Everything transferred with a standard > > RETR command should be accounted for." > > The sentence following the above sentence in the draft states: > > "This means that top headers, followed by messageparts will be > transferred, in that order." Yes, but I didn't read it as "byte-stuffing will not affect offsets", the "everything" seemed to imply the opposite. > > I am assuming the "everything" applies to byte-stuffing. > > Read the above quote. So it needs to be clearer. > > The question: what offset should it request this time? It could have > > stored the amount of data it has received, along with the incomplete > > message. However, that amount is one octet larger than what it would > > be if the transfer wasn't interrupted the first time. Obviously, the > > server, according to this definition of the extension, expects to see > > offsets into the data that would be "transferred with a standard RETR > > command" -- that is, without any previously recovered transfers. > > OK, this is something I thought about when I was drafting the memo. I > decided not to include an explanation because I saw it as obvious. > Having given it some more thought, I think a more thorough explanation > and an example might be in order. I will probably add a section > explaining how byte-stuffing and calculation of offsets should be done. Yes, it seems we need that. There's one more issue that you'd need to decide on -- are we calculating CRLF's the way they're transferred over POP3 (would be more essential for the protocol), or the way line separators are typically stored on a UNIX-based server (to allow for a simple seek). > The model should be very simple, please correct me if I'm wrong: > > msg-offset-bytestuff-server-client-bytedestuff-msg Unfortunately, there may be a few more conversions: server_mailbox -> skip(*)_to_offset -> bytestuff -> LF_to_CRLF -> ... ... transfer -> bytedestuff -> convert_to_client_mailbox_format (*) we haven't defined the "skip" for the server_mailbox "layer", yet. > Example: > > Message number is '1'. > Message size is 217,743 bytes. > Offset 113,435 starts with a '.'. > > Lets assume our transfer got interrupted after 113,434 bytes. Now, the To illustrate the case of having to byte-stuff, you'd want the transfer to be interrupted after 113,435 bytes. > received data has been bytedestuffed and saved, thus the client executes > 'RETR 1 113435'. The server opens the message and jumps to this offset. > It now bytestuffs the '.' as usual and transmits the data to the client > who bytedestuff the data and adds it to the incomplete data it received > earlier. > > > " a) Less consumption of bandwidth > > b) Less resources used on the server-side > > c) Less time used online for the client" > > > > I would consider (c) the most important reason. Bandwidth isn't that > > important here, as POP3 transfers are typically within one ISP, from > > a mailbox at the ISP to the ISP's dialup line, which is busy anyway. > > You have a point about bandwidth, in some cases, not being a problem. > However, taking in account the fast evolution on the internet, this > might in the future apply to other connections than modems, i.e. a > company's internal network, since attachments might be much larger than > they are today. Yes, that's a valid point as well, but, in my opinion, (c) is more important. > > (b) is not fully achieved with your definition of the offsets, as the > > server has to read the entire message from the mailbox to account for > > byte-stuffing. > > Well, this shouldn't be a problem with the above explained model. The > server just need to do an fseek() to the specified offset. I wish it was that simple. To allow for a simple seek, we would need to define that offsets are into the message as stored in a BSD-style mailbox. (I am assuming that performance is less critical on non-UNIX servers, and that they can choose the mailbox format that allows for better performance if they care.) The calculation of offsets on the client end would get more complicated, then -- unless the client uses single LF's for its mailboxes as well. Maybe it's not that bad, after all. Signed, Solar Designer Received: by ns.secondary.com (8.9.3/8.9.3) id TAA24770 for ietf-pop3ext-bks; Sun, 30 Jan 2000 19:15:54 -0800 (PST) Received: from musse.tninet.se (musse.tninet.se [195.100.94.12]) by ns.secondary.com (8.9.3/8.9.3) with SMTP id TAA24766 for ; Sun, 30 Jan 2000 19:15:52 -0800 (PST) Received: (qmail 8166 invoked from network); 31 Jan 2000 04:17:59 +0100 Received: from garibaldi.tninet.se (195.100.94.103) by musse.tninet.se with SMTP; 31 Jan 2000 04:17:59 +0100 Received: from rundegren.com (sdu52-203.ppp.algonet.se [195.163.203.52]) by tninet.se (BLUETAIL Mail Robustifier1.1.7) with ESMTP id s3274763t423128-bmr-garibaldi ; Mon, 31 Jan 2000 04:17:57 +0100 Message-ID: <3894FEF4.1D61AFF7@rundegren.com> Date: Mon, 31 Jan 2000 04:18:12 +0100 From: Anders Rundegren X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: Solar Designer CC: ietf-pop3ext@imc.org Subject: Re: Extended RETR - Draft posted References: <200001300344.GAA04323@false.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: > "The offset is given in octets. Everything transferred with a standard > RETR command should be accounted for." The sentence following the above sentence in the draft states: "This means that top headers, followed by messageparts will be transferred, in that order." > I am assuming the "everything" applies to byte-stuffing. Read the above quote. > The question: what offset should it request this time? It could have > stored the amount of data it has received, along with the incomplete > message. However, that amount is one octet larger than what it would > be if the transfer wasn't interrupted the first time. Obviously, the > server, according to this definition of the extension, expects to see > offsets into the data that would be "transferred with a standard RETR > command" -- that is, without any previously recovered transfers. OK, this is something I thought about when I was drafting the memo. I decided not to include an explanation because I saw it as obvious. Having given it some more thought, I think a more thorough explanation and an example might be in order. I will probably add a section explaining how byte-stuffing and calculation of offsets should be done. The model should be very simple, please correct me if I'm wrong: msg-offset-bytestuff-server-client-bytedestuff-msg Example: Message number is '1'. Message size is 217,743 bytes. Offset 113,435 starts with a '.'. Lets assume our transfer got interrupted after 113,434 bytes. Now, the received data has been bytedestuffed and saved, thus the client executes 'RETR 1 113435'. The server opens the message and jumps to this offset. It now bytestuffs the '.' as usual and transmits the data to the client who bytedestuff the data and adds it to the incomplete data it received earlier. > " a) Less consumption of bandwidth > b) Less resources used on the server-side > c) Less time used online for the client" > > I would consider (c) the most important reason. Bandwidth isn't that > important here, as POP3 transfers are typically within one ISP, from > a mailbox at the ISP to the ISP's dialup line, which is busy anyway. You have a point about bandwidth, in some cases, not being a problem. However, taking in account the fast evolution on the internet, this might in the future apply to other connections than modems, i.e. a company's internal network, since attachments might be much larger than they are today. > (b) is not fully achieved with your definition of the offsets, as the > server has to read the entire message from the mailbox to account for > byte-stuffing. Well, this shouldn't be a problem with the above explained model. The server just need to do an fseek() to the specified offset. // Anders Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id TAA28135 for ietf-pop3ext-bks; Sat, 29 Jan 2000 19:45:34 -0800 (PST) Received: from mystic.false.com (root@false.com [198.65.171.171]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id TAA28131 for ; Sat, 29 Jan 2000 19:45:32 -0800 (PST) Received: from false.com (root@false.com [198.65.171.171]) by mystic.false.com (8.9.3/8.9.3) with ESMTP id VAA04905; Sat, 29 Jan 2000 21:47:23 -0600 Received: (from solar@localhost) by false.com (8.8.5/8.8.5) id GAA04323; Sun, 30 Jan 2000 06:44:35 +0300 From: Solar Designer Message-Id: <200001300344.GAA04323@false.com> Subject: Re: Extended RETR - Draft posted In-Reply-To: <3892DE6A.CD19311@rundegren.com> from Anders Rundegren at "Jan 29, 0 01:34:50 pm" To: anders@rundegren.com (Anders Rundegren) Date: Sun, 30 Jan 2000 06:44:35 +0300 (MSK) Cc: ietf-pop3ext@imc.org X-Mailer: ELM [version 2.4ME+ PL31 (25)] MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: > The draft for the extended RETR command has now been posted. Feedback > is, as always, very appreciated. I'm not sure about the section dealing > with extended response codes, it definitely needs some more work. Since > English is not my native language I ask you to look over this too. > > http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt The initial impression is that this clearly needs more work, perhaps resulting in a different approach to define offsets within a message. "The offset is given in octets. Everything transferred with a standard RETR command should be accounted for." I am assuming the "everything" applies to byte-stuffing. Now, let's assume a transfer got interrupted right before a dot in the message. The client requests that the transfer is continued from that offset, and the operation succeeds. The server byte-stuffs that dot (as it is now at the beginning of a line being transferred), and transfers more of the message to the client. Unfortunately, the client gets disconnected once again. It wants to recover the transfer. The question: what offset should it request this time? It could have stored the amount of data it has received, along with the incomplete message. However, that amount is one octet larger than what it would be if the transfer wasn't interrupted the first time. Obviously, the server, according to this definition of the extension, expects to see offsets into the data that would be "transferred with a standard RETR command" -- that is, without any previously recovered transfers. With this definition of the RETR extension, the client would have to detect if the first character it receives after a transfer continues is a dot _and_ it is not at the beginning of a _message_ line. It should then subtract one from the data size it is calculating, for the case of having to continue the transfer once again. This should be at least documented in the same RFC, but I suggest that we better avoid this extra complexity somehow. A more obvious requirement of this definition is that the client either stores the amount of data it has received for each incomplete message, or re-calculates that amount once it wants to finish the transfer. The former is fine if we only want to recover transfers while the client process is running, but might require extending mailbox format for some clients if we want that functionality even after an OS restart. The latter puts some code from POP3 servers, which used to be server-specific, into clients as well. Finally, speaking of your reasoning for the extension -- " a) Less consumption of bandwidth b) Less resources used on the server-side c) Less time used online for the client" I would consider (c) the most important reason. Bandwidth isn't that important here, as POP3 transfers are typically within one ISP, from a mailbox at the ISP to the ISP's dialup line, which is busy anyway. (b) is not fully achieved with your definition of the offsets, as the server has to read the entire message from the mailbox to account for byte-stuffing. Signed, Solar Designer Received: by ns.secondary.com (8.9.3/8.9.3) id EAA13788 for ietf-pop3ext-bks; Sat, 29 Jan 2000 04:32:51 -0800 (PST) Received: from musse.tninet.se (musse.tninet.se [195.100.94.12]) by ns.secondary.com (8.9.3/8.9.3) with SMTP id EAA13784 for ; Sat, 29 Jan 2000 04:32:49 -0800 (PST) Received: (qmail 20401 invoked from network); 29 Jan 2000 13:34:48 +0100 Received: from garibaldi.tninet.se (195.100.94.103) by musse.tninet.se with SMTP; 29 Jan 2000 13:34:48 +0100 Received: from rundegren.com (sdu252-203.ppp.algonet.se [195.163.203.252]) by tninet.se (BLUETAIL Mail Robustifier1.1.7) with ESMTP id s2846419t372929-bmr-garibaldi ; Sat, 29 Jan 2000 13:34:46 +0100 Message-ID: <3892DE6A.CD19311@rundegren.com> Date: Sat, 29 Jan 2000 13:34:50 +0100 From: Anders Rundegren X-Mailer: Mozilla 4.7 [en] (X11; I; Linux 2.2.12-20 i686) X-Accept-Language: en MIME-Version: 1.0 To: ietf-pop3ext@imc.org Subject: Extended RETR - Draft posted References: <182081.3157782802@nifty-jr.innosoft.com> Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: 7bit Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: The draft for the extended RETR command has now been posted. Feedback is, as always, very appreciated. I'm not sure about the section dealing with extended response codes, it definitely needs some more work. Since English is not my native language I ask you to look over this too. http://www.ietf.org/internet-drafts/draft-rundegren-pop3-00.txt Thanks! // Anders Rundegren Received: (from majordomo@localhost) by ns.secondary.com (8.9.3/8.9.3) id MAA14532 for ietf-pop3ext-bks; Sun, 9 Jan 2000 12:32:04 -0800 (PST) Received: from eljefe.innosoft.com (ELJEFE.INNOSOFT.COM [192.160.253.137]) by ns.secondary.com (8.9.3/8.9.3) with ESMTP id MAA14528 for ; Sun, 9 Jan 2000 12:32:03 -0800 (PST) Received: from nifty-jr.innosoft.com ([192.160.253.35]) by mail.eljefe.innosoft.com (PMDF V6.0-17 #4409) with ESMTPA id <0FO3001654LYAJ@mail.eljefe.innosoft.com> for ietf-pop3ext@imc.org; Sun, 9 Jan 2000 12:22:50 -0800 (PST) Date: Sun, 09 Jan 2000 12:30:58 -0800 From: Chris Newman Subject: Re: POP3 "resume download" extension In-reply-to: <199912170622.BAA02584@astro.cs.utk.edu> To: Keith Moore , vrs@rampnet.com Cc: Anders Rundegren , "ietf-pop3ext@imc.org" Message-id: <365737.3156409858@nifty-jr.innosoft.com> MIME-version: 1.0 X-Mailer: Mulberry (MacOS) [2.0.0b5, s/n S-100003] Content-type: text/plain; charset=us-ascii; format=flowed Content-transfer-encoding: 7bit Content-disposition: inline Sender: owner-ietf-pop3ext@imc.org Precedence: bulk List-Archive: List-ID: List-Unsubscribe: --On Friday, December 17, 1999 1:22 -0500 Keith Moore wrote: > fwiw, a lot of folks are likely to push back on defining any new extension > to pop. there is a feeling that two mail access protocols are too many, Let me explain my theory for why we can't get rid of POP easily. The primary customer for POP servers is large ISPs providing individual user service. These ISPs need to put as many mail users on a server as possible in order to keep management costs to a minimum so the per-user margin is profitable. Most of the server operating systems which can scale large enough with enough reliability and performance are based on the BSD-socket API. When machines get that big, two major limiting factors on scalability are a small fixed limit on TCP/IP connections (max file descriptors) and the exponential performance cost of select and poll. In this scenario, TCP/IP connections are expensive and need to be kept as short lived as possible. IMAP is anethema, as would be any extension to POP which might result in longer-running connections. The excessively ugly /dev/poll interface in Solaris 8 (pre-release) fixes half the problem but that OS still has a small hard-coded limit on the number of file descriptors instead of making RAM the only limiting factor as it should. So we have to live with protocols that use short-lived TCP/IP connections like POP and HTTP 1.0 (with the shared Internet suffering as a result). > if you do define an extension for partial message retrieval I think it > should allow partial message retrieval (e.g. byte count) rather than > just resume. that way, you don't have to drop a connection to > get the server to stop sending you a message. I happen to support the idea of a resume download extension, but I'm opposed to adding a partial message retrieval extension to POP. My reasons are subtle (mostly gut reaction), but I'll attempt to explain: First, I don't believe most clients would bother to do a pipelined chunking download of POP mail. It simply adds too much complexity to the common case (download all mail and delete from server) in order to optimize an uncommon case (user cancel of large message). Simply closing the socket and re-opening is much less error prone, and wasting a few round-trips simply isn't a performance concern when you're interacting with the user. In IMAP, connections are usually long-lived so a pipelined chunking download makes more sense, but even though the facility is present in the protocol it's largely unused. Furthermore, the interactions between dot-stuffing and byte ranges would make the POP3 version tricker to implement than the IMAP version. That's far too much complexity to save a few round trips. Second, I don't think POP3 is the right protocol to start experimenting with new user cancel facilities. Most of the times the IETF has tried an explicit user cancel facility in a protocol, the result has largely failed (e.g., telnet TCP urgent data & IMAP pipelined/chunked partial fetch). A largely stable full standard protocol is the wrong place to experiment. Finally, I think a byte-range fetch could be abused. In particular, some clients will try to use it to fetch pieces of a large message (something which IMAP does far better). This would require a series of round trips to see if enough data has been fetched. So if you're going to add a byte-range fetch, you also need to add a separate facility to at least fetch to the end of the first text part (possibly with a size limit as well) at the same time to minimize the potential for abuse. But now the extension adds one or two commands and a bit of server-side MIME parsing which is really pushing the complexity barrier for a POP extension. A "resume download" extension is simpler and doesn't have the above issues. About all it can be used for is to reduce retransmission of data after a user cancel or network failure -- thus making a POP client signficantly more likely to succeed at downloading a large message over a flaky connection. That has an acceptable complexity:benefit ratio in my book, byte-range fetch does not. - Chris