]> Dell

300 Innovative Way Nashua NH 03062 US +1 781 572 8038 davenoveck@gmail.com

Transport NFSv4 This document surveys the processes by which the NFS protocol has been extended in the past, including all of the NFS major and minor versions. A particular focus is on how the minor versioning model of NFSv4 has worked and what might be done to enhance version management within NFSv4. The work already done in the new NFSv4 version management document is summarized, and there is a discussion of further issues the working group will need to address in moving that work forward.

This document examines the subject of protocol extension within the NFS family of protocols. In order to better understand the issues that exist going forward with NFSv4, we examine the history of protocol extension throughout the development of NFS including The pre-IETF period The development of successive NFSv4 minor versions, under minor versioning rules first defined in and subsequently modified based on the working group's then-current needs. The consolidation of minor versioning rules in along with the other changes made in that document to make NFSv4 versioning more flexible. With this history in mind, we discuss the issues that the working group needs to address to create an NFSv4 versioning RFC that will serve to define a comprehensive approach to version management for the NFSv4 family of protocols.

It is intended that these key words be used in this document in a way consistent with their definition in . However, because of the considerations noted below, there are some issues that readers need to be aware of: It is not altogether clear whether these keywords are to be limited to requirements regarding protocol implementations, or whether they can reasonably be used in specifying guidance directed at future potential RFCs. The nature of this document is such that it does not directly specify implementation requirements. In some cases it discusses potential requirements that an RFC derived from ) might direct to implementations. In other cases, requirements are one step farther removed from protocol implementations. For example, the versioning RFC might specify the kinds of requirements that future minor version definition documents and feature definition documents might specify. The documents that are discussed herein, have at times used these key words in a way that seems to be at variance with the definitions in . For details, see Sections and . The reader should be aware of our basic practices regarding these terms. We only apply these upper-case key words to directions regarding protocol implementations, rather than to guidance intended for those writing RFCs. As a result, these keywords will often appear as quotations from existing documents, either direct or indirect. Readers should be aware that it is possible that some such uses might not be in accord with . In other contexts, these keywords will be in the context of proposals/suggestions of what future RFCs could or might say regarding certain issues. The use of these terms as part of minor versioning rules poses troublesome issues in the context of this document. These rules have appeared in , , and , and also in various drafts of and . During this period there were multiple switches between the RFC2119 keywords and corresponding lower-case terms. Use of the terms "OPTIONAL" and "REQUIRED" for feature status would conform to our general practice since a feature status is essentially a way in which a minor version specification RFC might REQUIRE (or not) implementations to support a given feature. Unfortunately, "RECOMMENDED" would not fit that model very well. In this document, in the interests of uniformity, we will use lower-case terms for feature statuses. except when we are referring to what is said in a particular document which used the upper-case keywords.

Protocols often require means by which they can be extended. Such extensions may be needed to meet new requirements, to correct protocol weaknesses exposed by experience, or even to correct protocol bugs (as can happen when protocols are published as RFC's without fully fleshed-out implementations). We need to distinguish here between protocol "extension" and "versioning". Versioning is a form of protocol extension but not every form of protocol extension can be accommodated within a versioning paradigm. When a versioning paradigm is in place, groups of extensions are conceived of as ordered, allowing extensions in subsequent versions to build upon those in previous versions. When multiple extensions are combined into a single version, each of the extensions may be built assuming that the others will be present as well. In such cases, there can be the opportunity to make design changes in the protocol, allowing elements of the protocol to be restructured, sometimes in major ways. When a versioning paradigm is in effect and extensions are optional, extensions cannot build upon one another, since the presence of any particular extension cannot be assumed. In such cases, the ability to restructure the protocol is reduced, but smaller changes may be introduced more easily. In this latter case, it is not clear that the word "versioning" is appropriate. Nevertheless, in this document, we will, as in the phrase "NFSv4 minor versioning" use the existing terminology without necessarily subscribing to the view that "versioning" is the appropriate description.

Some factors that are often relevant in deciding on the means by which a protocol will be extended. Whether extensions are to be individually selectable (i.e. optional) or assumed to be always present, allowing one to build upon another earlier one. The size and scope of extensions that will be made. Compatibility issues with existing implementations. Issues that relate to ensuring that when individual extensions, separately arrived at, are each optionally allowed, that the ones used are compatible and can be used effectively together. The overall implementation framework. For example, RPC-based protocols may do extension by means of the RPC version mechanism. While it is possible to use different sorts of extension mechanisms for different sorts of extensions, protocols typically do not take advantage of that flexibility. On the other hand, protocols do, as NFS has done, change their preferred extension mechanisms in response to long-term changes in requirements. However, once having done so, they rarely switch back. Changing extension mechanisms is a big step, both conceptually and in implementation terms, and is not frequently repeated.

Often, protocols will be designed with specific mechanisms, designed to allow protocol extension. An example is the provision for TCP options (see and .) Most often, such mechanisms are designed to allow individual extensions to be designed and implemented independently, with any dependency relations between extensions specified separately and not enforced by the extension mechanism itself.

RPC-based protocols may, and often do, provide for protocol extension by replacing the XDR for one version with that for another and using the RPC versioning mechanism to manage selection of the proper protocol variant. The use of the RPC versioning mechanism enforces a versioning paradigm of this sort on protocols using this extension mechanism. This extension mechanism allows very extensive protocol changes, up to and including the replacement of one protocol by an entirely different one. For some kinds of protocol extensions, this seems the only way to effect the change.

It is possible to replace an XDR definition by one which is an extension in the sense that The set of messages described by the second definition is a superset of that described by the first. Each message within the set of messages described by the first definition is recognized as having exact the same structure/ interpretation by the second definition. Each message within the set of messages described by the second definition but not the first definition must be recognized as part of an unsupported extension. Within an XDR/RPC framework, extensions can be arrived at by: Addition of previously unspecified RPC operation codes. Addition of new, previously unused, values to existing enums. Addition of previously unassigned bit values to a flag word. Addition of new cases to existing switches, provided that the existing switch did not contain a default case. Such an extension relation between XDR descriptions is reflexive and transitive and thus defines a partial order. In practice, provisions have to be made to make sure that two extensions of the same description are compatible (i.e. either one is an extension of the other, or there is a another description that is a valid extension of both). To put things in concrete terms, such compatibility can be assured if measures are taken to ensure: That the same operation code is not used for two different RPC operations. That the same enum value is not assigned two different meanings. That the same bit flag value is not assigned two different meanings. That whenever the same case value is added to the same switch in two different extensions, the content assigned to the two matching added cases is the same. That default cases are never added to existing switches.

It is possible to use multiple of these means of protocol extension simultaneously. When this is done, the result is a composite extension mechanism. For example, if the XDR replacement or XDR extension mechanism is adopted, a protocol-specific mechanism can be added to it, if the protocol-specific mechanism is built on objects whose XDR definition is sufficiently generic. (e.g. opaque arrays or feature bitmasks). It can be argued that the NFSv4 attribute model provides such an embedded protocol-specific extension mechanism, since sets of attribute values are specified as XDR opaque arrays and attribute sets are specified as variable-length arrays of 32-bit integers allowing new attribute bits to be defined outside of the XDR definition framework. Note that there exists specification text that suggests that attributes are part of the XDR specification, making it hard to reach a firm conclusion on the matter. However, the resolution of this question does not affect the other matters discussed below, since, in either case, we have an extension mechanism that allows optional extensions.

NFSv2 and NFSv3 were described by the informational RFC's and . These documents each described existing interoperating client and server implementations. Thus they started with running code. If there was a rough consensus in effect, it was that these were useful protocols to use and thus that someone building a client or server had to interoperate with the existing implementations. The following characteristics of protocol development during that period are noteworthy. Most client implementations were implemented on very similar systems, in terms of the API's supported and many specifics of the local filesystems exported by servers. Often, the important client and server implementations were done by the same organization. As a result of these commonalities, specifications tended to avoid a lot of detail that would have been required in a more diverse environment. New features were thought of in terms of generally understood client and server implementation frameworks and it was generally clear which of those could be implemented without markedly changing that framework.

There were a number of significant changes involved, but only the first two were of major importance. Converting file sizes and offsets from 32-bit to 64-bit unsigned integers. Support for uncommitted WRITEs and the COMMIT request. Provision for WRITE to return atomic pre- and post-write file attributes, in order to allow a client to determine whether another client was writing the file. Interestingly enough, this feature was not carried over into NFSv4. The READDIRPLUS request. The addition of NFS3ERR_JUKEBOX (the precursor of NFS4ERR_DELAY). Of these only the first actually needed something as drastic as the XDR replacement model. The others could have been handled simply by adding new RPC requests and an enum value to an existing NFSv2 XDR. Since, NFS's extension mechanism was then XDR replacement, such choices were not available.

NFSv4 was the first NFS version published as a Standards track document. Although an initial , entitled "NFS version 4 protocol" was published as a Proposed Standard, it was never implemented due to issues with the design of locking. Subsequently, , entitled "Network File System (NFS) version 4 Protocol" was published as a Proposed Standard, obsoleting . Currently, there are bis documents, and , nearing publication. The set of changes made to create NFSv4 was larger by far than that for NFSv3. A partial list follows. Conversion to a stateful protocol, including support for locking. Locking facilities included support for OPENs (with share reservations), byte-range locking (optionally including mandatory byte-range locks) and delegations. The COMPOUND operation. Conversion to a bi-directional protocol, by the addition of (optional) callbacks. Internationalization. Support for filesystems doing case-insensitive name matching. A new, extensible file attribute model, including support for acls, and conversion of user and group to a string model, opening the way for multi-domain support. Support for named attributes. Merger of ancillary protocols (e.g. MOUNT, NSM, NLM) into the NFS Protocol proper. Support for crossing of filesystem boundaries within a server's file name space (originally done for the incorporation of MOUNT functionality). Support for such multi-server operations as migration, replication, and referral. Referrals were not explicitly mentioned in and are explained in . Creation of a minor versioning model (to be discussed in ) to allow further protocol extension. These features/extensions were implemented via the XDR replacement model. This was the only realistic alternative, not only because of the size of the list, but also because some of the changes undercut some of the central design elements of the pre-IETF NFS protocol.

The minor versioning approach for NFSv4 uses an XDR extension model. It was presented within a versioning paradigm but the fact that all the added features were to be (at least initially) optional indicated that features were intended to be built independently, and that clients were expected to deal with their presence or absence. Note that the term "features" is not explicitly defined. We assume that the definition includes operations within COMPOUND or CB_COMPOUND, attributes, added flag bits and enum values, and new cases of XDR switch definitions. Now let's look at some specifics of the minor version rules established for NFSv4 in . Note that some of these were significantly modified by and , as discussed in . No RPC requests may be added. Thus COMPOUND (and NULL) are to be the only requests within all NFSv4 minor versions. Similarly for callbacks, CB_COMPOUND and CB_NULL are the only requests within the callback program. The arguments for COMPOUND and CB_COMPOUND contain a 32-bit minorversion field. Although this field is part of the minor versioning paradigm, it is not clear how useful it is, as long as all extensions are optional. For a more detailed discussion of this issue, see . Features may be defined as optional, recommended, or required. Later, these designations were converted to use the corresponding upper-case terms defined in . See Sections and for details. These designations apply to implementation by the server. For clients, no operations are defined as required, although it is hard to imagine an NFSv4.0 client that does not use PUTFH or SETCLIENTID, for example. Features may be upgraded or downgraded along the optional/recommended/required scale. Features may be declared "mandatory to not implement". This allows the deletion of a feature while retaining as reserved the value previously assigned. Clients and servers that support a particular minor version must support all previous minor versions as well. New features may not be designated as required in the minor version in which they are introduced. Clients are not allowed to use stateids, filehandles, or similar returned objects from the COMPOUND procedure with one minor within another COMPOUND procedure with a different value of the minorversion field. This model was subsequently modified in and in . See Sections and for details. Many of the events anticipated in the model presented above have never been realized and it may be that they never will be realized. See for some details. Examples are: There have never been recommended operations. There have never been optional attributes. Features have never been upgraded or downgraded in the transition between minor versions.

NFSv4.1 was described in and , each of which was published as a Proposed Standard. NFSv4.1 made a major change to NFSv4.0. It was able to do primarily using an XDR extension model although it did not follow the rules laid out in . Instead, presented its own set of minor versioning rules. The following major changes in the rules were made. Uses of the terms "recommended", "required", "should", etc. were switched to use the corresponding upper-case words defined in . This included conversion of "recommended" attributes to be "RECOMMENDED". The reasons for this change remain unclear. Unlike the changes noted below, this change was incorporated in . For a discussion of how this situation was dealt with during the IESG review of that document, see The rule requiring that features be introduced initially as optional was modified to enable features described as "infrastructural" to be required upon introduction. Also, the requirement that clients and servers support previous minor versions changed from a "must" to a "SHOULD". Presumably, this change reflects the fact that a minor version with substantial infrastructural changes is essentially a new protocol, making the "must" seem dubious. Whether the "SHOULD" here is in line with needs to be explored. NFSv4.1 also bypassed the versioning rules by making non-XDR-based protocol changes. See for a discussion of the logic behind such changes. The following features were added as infrastructural features. Support for a sessions model including support for EOS (exactly-once semantics). Implementation of this feature included some non-XDR-based changes in operation behavior. Note that COMPOUND was taken advantage of to avoid adding slot and sequence information to the request header. Instead this information is packaged in a SEQUENCE or CB_SEQUENCE operation at the start of the COMPOUND or CB_COMPOUND. A new set of operations were added which enable the client and server to identify themselves to one another. Although these are often thought of as part of the sessions model, in fact they are logically distinct. RECLAIM_COMPLETE was added to allow better server sequencing of lock reclaim operations. In addition to these required features, there were a number of non-XDR-based changes made in NFSv4.1. Although not thought of as "features" at the time, they are described in , which gives no indication that support is optional. These include: Addition of a new special stateid to represent the current stateid. New rules for the interpretation of a stateid seqid of zero. New rules regarding the interaction of the MODE and acl-related attributes. There also a number of non-required features. While such features are optional in the sense that a server may or may not support them, it is not clear that all are optional in terms of the existing minor versioning rules. Given that all attributes are specified as REQUIRED OR RECOMMENDED, it may be that attributes that may or may not be supported are considered recommended from the viewpoint of the minor versioning rules. Here and in the future we will use "non-required" instead of "optional or recommended". The following non-required features were added. Parallel NFS WANT_DELEGATION to allow delegations to be obtained apart from opens. Directory delegations and notifications. The MODE_SET_MASKED, SACL, DACL attributes. The FS_LOCATIONS_INFO and FS_STATUS attributes. Note that there has been little implementation work on the last two of these. Parallel NFS created an alternate protocol extension mechanism for NFS. New pNFS mapping types could be added. Existing mapping types might have their own extension mechanisms. There also exists the possibility that features might be added within the NFSv4 protocol proper, designed to, or capable of, interacting with particular mapping types. This document will not these issues but eventually, the NFSv4 Versioning RFC will have to address them.

While NFSv4.2 has not been defined in an RFC, its development has been going on for some time and it is unlikely to experience additions to or deletions from its feature set before completion. The descriptions in and can serve as useful references for this analysis. This is despite the fact that some of the features may later experience changes. Because of the lengthy process involved in producing NFSv4.1, the working group decided that NFSv4.2 was to be a relatively small minor version consisting only of optional features which would be documented by specifying changes from NFSv4.1. The following features (all optional) are provided for in NFSv4.2: Support for labeled NFS. Server-side copy. An operation fence option on EXCHANGE_ID. Application data holes (formerly application data blocks). Disk-space reservation (nominally "recommended" since it is implemented by an attribute and attributes have never been declared "optional"). Hole-punching operations. READ_PLUS. Note that there are two pieces of infrastructure that are used by multiple features above. These are not "infrastructural" in the sense mentioned in (i.e. they are not required), but they do serve an infrastructural role in that are required to be present if one of the optional features that use them are supported. WRITE_PLUS used to implement (ordinary) hole punching and application data holes. OFFLOAD operations/callbacks used to support WRITE_PLUS and server-side copy.

As noted above, there have been changes made by and in the NFSV4 minor versioning model. NFSv4.1 (in ) introduced the concept of "infrastructural" features (i.e. those defined as required at initial introduction). NFSV4.1 made additional non-XDR-based changes, which, although not explicitly defined as such, were effectively required. Taking note of these changes, we can classify potential minor versions, starting with those that currently exist, based on how they affect inter-version compatibility. Minor version zero which introduced a new (major) version of the NFS protocol. All of the operations within it are new and a subset are effectively required. Minor versions which make required changes in the protocol that affect all implementations of the previous minor version. Such changes may include addition of required/infrastructural feature, incorporation of an effectively required non-XDR-based change, or the deletion of a required feature by making it mandatory to not implement. Currently, the only such version is minor version one, although it is possible that there could be others in the future. Features that only add optional/recommended features, each present or absent on the server with clients needing to be able to deal individually with their presence or absence. Currently, the only such version is minor version two. It is likely there will be others in the future and it is possible that all future minor versions will be of this character. Minor versions which make required changes in the protocol that will affect some implementations of the previous minor version. These can result from feature status changes. Changing a non-required feature to required affects only implementations that do not support the feature. Conversely, deleting a non-required feature by making it mandatory to not implement. affects only implementations that do support the feature. No such minor versions currently exist.

Minor versions which may affect inter-version compatibility, form an ascending sequence in which we also have a versioning paradigm, implemented principally using XDR extension. Optional-feature-only versions are fundamentally different. Each NFSv4.2 server implements the same protocol as NFSv4.1 with a particular set of optional or recommended features beyond those that are required. This set may range from the null set all the way to all of the non-required features. Here, it appears that the versioning paradigm is not appropriate to the reality of the extension mechanism. As a way of illustrating the basic point here, let us consider two servers each of which only supports operations within NFSv4.1: The first server "supports" NFSv4.2 but none of the non-required features added in . In this case, any attempt by a client to use one of those features will result in an NFS4ERR_OPNOTSUPP being returned. Let us say that the second server does not support NFSv4.2 and supports precisely the same set of features. In this case, a request will be rejected (with error NFS4RR_MINOR_VERS_MISMATCH) if its COMPOUND minorversion field is two and if the field is one, any unsupported NFSv4.2 operation will be rejected with NFS4ERR_OP_ILLEGAL. Although this obeys the rules as they stand, there is no obvious value for the client, the server, or the protocol in making these artificial distinctions. Optional-feature-only minor versions such as NFSv4.2 are not minor versions in the same sense that NFSv4.0 and NFSv4.1 are. In this case the minorversion field is not providing any information, while the set of operations supported is the important thing that the server implementer chooses and the client needs to know. It is not clear how thus mismatch will be resolved. Nevertheless, it is an indication that the focus previously placed on the construction of minor versions was misplaced and that managing the addition of features is in most cases the fundamental issue. How minor versions are to fit within that framework is something that the group needs to decide.

To summarize protocol extension as it has been applied to the NFSv4 protocols: NFSV4.0 was implemented using the XDR replacement approach inherited from NFSv2 and NFSv3. As was to be expected given the nature and scope of the changes, its development took considerable time. It defined a protocol extension approach based on the XDR extension mechanism which was designed to enable future development of minor versions. However, this mechanism was not used as part of the implementation of NFSv4.0. NFSV4.1 was primarily implemented using the XDR extension mechanism. To implement sessions, it was forced to modify the extension approach in the only way that was viable in the circumstances. As a result, the specification process took a long time, since it made significant structural changes to the protocol and also because it specified the entire protocol in a new RFC, rather than documenting a set of extensions. NFSv4.1 also made a number of modifications to the NFSv4 protocol which did not involve any XDR-based changes at all. These are discussed in . While not considered infrastructural, or even thought of as "features" at the time, these changes are effectively required and the possibility of such changes needs to be considered as the working group formulates its approach to the problems of NFSv4 version management. NFSV4.2 returned to the original XDR extension mechanism and was intended to be a small incremental update with a one-hundred page (or less) specification. The fact that this turned out to be a multi-year effort has occasioned concern and we will discuss how the process can be streamlined and otherwise made more effective. The history of NFSv4.2 development serves to illustrate some of the inherent problems in the then-current approach to minor versioning. Since similar problems can be expected to recur unless that approach is changed, attention needs to be paid to understanding why such difficulties were experienced. It is important to note that NFSv4.0 (in and ), both about 300 pages and NFSV4.1 (in ), about 600 pages documented the entire minor version. On the other hand, the NFSv4.2 specification document (in ) simply specified the changes from the previous minor version, in about 100 pages. Given the difficulties of writing very large specifications, it has to be considered extremely unlikely, that any future minor versions will be documented other than as a set of changes from the previous minor version.

Although not recognized at the time, the existence of non-XDR-based protocol changes is part of the existing NFSv4 versioning approach and must be addressed in any revision of NFSv4 versioning. Such changes have been of two types: Changes in field in interpretation and use. Although the intention has always been that all such matters were to be defined in XDR, there are areas where this is not true. The interpretation of certain opaque fields and of strings are cases in which the field interpretation and use is defined in protocol specification text. Changes in operation semantics. These may apply to a few operations or to most or all operations. All such changes have been implemented as required with implementations using the minor version field of the COMPOUND to determine whether the change applies to a particular request.

The following pattern was followed for NFSv4.2, and, unless changes are made, seems likely to persist. Various features are sketched out in individual drafts The working group reaches a decision (i.e. by rough consensus) as to the extensions/features to be included in the minor version. At the time this decision is made, the features may vary as to their maturity. Some have individual drafts which sketch them out while others do not. Any existing individual drafts are combined into a draft of a working group document intended to eventually evolve into an RFC describing the new minor version. These are then supplemented with descriptions of the other features chosen for inclusion. This document goes though further refinement and cycles of working group document review. At some point a companion -dot-x document is prepared and reviewed by the working group as well. The two documents go through working group last call, IESG review, and RFC publication. This pattern of development is not a good fit for the kind of minor version that NFSv4.2 is and many future such minor versions will be. Such versions consist of a set of mostly unrelated features, each individually selectable or not by implementers, artificially yoked together. In essence, we have a "feature batch" rather than a minor version.

A number of issues have been noted with the existing process for NFSv4.2, leading to the conclusion that the process needs to be revised in some way for future minor versions, of the same sort. It takes too long to get a minor version drafted and through working group and IESG review. Despite the fact that NFSv4.2 was intended to be a fairly minimal minor version, describable in a one-hundred-page spec, it looks like the pace of development is such that there will be nearly a four-year delay from the time that the first NFSv4.2 draft was submitted to the point at which the final draft is ready to be submitted to the IESG. The timeline for some of the features within NFSv4.2 is even longer. It will have taken about six years for the inter-server copy feature to go from initial draft to IESG submission of the minor version of which it is a part. Only very recently have we had significant active implementations in which proposed last-minute protocol changes can be tested for validity. As an example of the problem, consider the decision to pass source stateids to the COPY op. If we had had prototype implementations of inter-server copy, the problems that this created (since stateids are tied to clientids in NFSv4.1 and beyond) would have quickly become manifest. Many features within NFSv4.2 did not received the kind of searching review appropriate to later stages of specification development. Some instances of problems/issues ascribable to a lack of searching document review are described below. Rather than requiring the necessary review prior to feature acceptance, a common pattern has been that important issues are only discovered on those occasions in which it appears that work on the minor version is coming to a close and that there are issues that have to be addressed before they create difficulties for prospective implementers. The state of the IO hints feature is most unsatisfactory. It is not clear how, or even if, it is possible to specify this in a way that interoperable clients and servers can be written which respond appropriately to such hints so that useful performance improvements can be demonstrated. It was the general understanding within the group that labeled NFS required use of RPCSEC_GSSv3, when in fact, only one case of labelled NFS, the server-enforced one, required it. When it became clear that RPCSEC_GSSV3 had not been worked on for a long time, the working group had to address that issue seriously. The security for inter-server copy was specified to be dependent on RPCSEC_GSSv3, yet, when it was found that RPCSEC_GSSv3 seemed not to be on the horizon, it turned out that a simple alternative was available. After a great deal of uncertainty about whether the functionality needed for inter-server copy would really be doable in RPCSEC_GSSv3, the working group had a range of choices for inter-server copy security. Later, after much work was done on RPCSEC_GSSv3, the working group has the option of going back to an approach dependent on RPCSEC_GSSv3. Application data blocks and related features have had a complicated history within NFSv4.2. Application data blocks were superseded by application data holes. There was little interest in implementing the feature since, as specified, a server implementation would require extensive filesystem extensions. Also, client-side API's were lacking, meaning that the only possible client-side implementations would be in special-purpose clients tightly bound to a particular implementation. Nevertheless, the feature continued in this unsatisfactory state for a long while. Eventually, it became clear that, as the feature was defined, it imposed significant implementation constraints even on NFSv4.2 clients not implementing the feature. At this point, it became clear that significant changes had to be made. This issue was resolved by limiting the scope of the proposed feature so that servers were not required to remember the parameters relevant to application data holes, but only the data that they represent. If we look at the problems above, we can understand better how such problems can arise. In short, the decision as to what features to include within a minor version, is not a good use of the rough consensus model and in proceeding on that basis, the group created a set of perverse incentives that undercut the process. Also, as the process goes on for a long time, as is likely, these perverse incentives are intensified. Consider the following points: It is not clear exactly what the consensus as to proposed minor version contents actually means. Working group members might interpret it as meaning "These features are worth pursuing and they should be pursued". However, if they thought the definition was more like, "each of these features is so important that, if it is not ready, any other feature, including the one I'm interested in, should be delayed also", then it is hard to imagine any such rough consensus existing. Note that, given the minor versioning implementation laid out above (in ), the latter definition is, for functional purposes, the effective meaning, of the minor version content consensus. Given that many features are linked together, any delay in one feature, once it is accepted as part of the feature batch, delays all of the features, making it hard for people to comment forthrightly on any significant specification inadequacies. Not only will it delay your preferred feature, but if the problems are not fixed, the only recourse is an extreme penalty. As a result, it often seems not worth pursuing these sorts of issues. As the version turnaround cycle is so long, it is very difficult to remove a feature from a minor version feature batch. Given that these are all features that have enough interest to be in the minor version, it is hard to transfer the feature into the next minor version, given that doing so will certainly result in a multi-year delay, at a minimum. As a result, features, once accepted, have an implicit guarantee of inclusion in the minor version, undercutting the motivation of the proposer and others to work to move the feature forward. On the other hand, uncertainty about the time of specification completion, often makes it is hard to plan for and allocate resources to development of client and server implementations. Given that responsibility for a minor version is transferred to the editor of the minor version definition document at an early stage, we have a process in which it is not clear who has responsibility to follow up on the work necessary other than the minor version editor who may not have the required time and expertise in all cases. There is not a specific feature owner with the responsibility to make the feature happen.

The following issues led the working group to consider formulation of a new approach to NFSv4 versioning: The experience of NFSv4.2 showed that, although we needed to make our documents smaller, doing so without other changes in how we did things left important issues unaddressed. The lack of implementation experience for many features led to a general feeling that the working group needed to do more to encourage/require development of feature implementations before they were published as Proposed Standards. The publication of multiple minor versioning rules came to seem to variance with the idea of a rule-based approach. Eventually, it was decided that a single version management document was needed for NFSv4 as a whole.

The working group has started work on a standards-track document () defining an approach to version management that is intended to apply to NFSv4 as a whole. Noteworthy advances in that document include Creation of a set of minor versioning rules to form a basis for a unified of set versioning rules. Such a set of rules would supersede the per-minor-version rules that had previously been present. Creation of the concept of extensible minor version with an associated workflow that avoids many of the problems noted above with regard to the batching of features. Explicitly allowing XDR changes to correct protocol errors. This addresses a situation in which there was sufficient uncertainty about such changes to prevent them from being considered, even though they were not explicitly disallowed.

During this period, changes were made to some related documents, as they moved toward publication. Some of these changes were made to simplify handling of the transition in versioning models that would occur upon publication of as an RFC. During the IESG review process, was modified to eliminate specific minor versioning rules. This made sense because the rules in question did not apply to NFSv4.0, and were, with regard to NFSv4.1 and beyond, superseded by those in . The restatement of minor versioning rules in was eliminated. In its place was left a short statement of v4.2-relevant exceptions from existing rules. The text is compatible with the base rules being taken from either or . As a result of these changes, upon publication of an NFSv4 version management RFC, it will only need to be marked as updating . Another relevant change concerned the use of the term "RECOMMENDED" regarding attributes which are not REQUIRED. As it appeared that this usage is inconsistent with , was modified to include the following statement: The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this document are to be interpreted as described in RFC 2119 except where "REQUIRED" and "RECOMMENDED" are used as qualifiers to distinguish classes of attributes as described in Section 1.3.3.2 and Section 5. A reasonable conclusion to draw from this is that while certain attributes are indeed REQUIRED, the other ones cannot be designated as "RECOMMENDED" as that term is defined in . The following issues remain to be resolved: How to deal with the analogous issue in . How this affects the use of "RECOMMENDED" (and possibly of "recommended") as a feature status.

As of now, the versioning document incorporates two conceptual models which need to be better integrated as the document moves toward RFC status The versioning rules inherited from are focused on minor versions and individual feature elements (which are referred to as "features"). Even though the addition of incremental features is now at the core of our prospective versioning practice, the rules relate to that only by implication. The version extensibility and protocol fix work is focused on (non-required) features, but has no real places for minor versions other than as a passive recipient of those features. It was decided to focus on this aspect of version management for good reasons but if the working group wants to leave open the possibility of minor versions that might refactor things or otherwise allow other sorts of minor-version-level change, the versioning RFC should be written to enable that.

With the exception of the incorporation of minor versioning rules (based on those in ), the current NFSv4 versioning document is focused on addressing the problems that have been seen with minor versions that are like NFSv4.2, in that they add non-required features and do nothing else. Such versions may not make any of the following types of changes: Addition of required features. Changes in the status of existing features, either to upgrade or downgrade them. In this context, eliminating a feature, i.e. making it mandatory to not implement, is considered a form of downgrading. Non-XDR changes in the protocol. Instances of such changes, which include changes in field interpretation and use and changes to operation semantics have occurred in NFSv4.1 and are discussed in . Such changes are not provided for in the existing minor versioning rules. Each of the above types of change has occurred as part of NFSv4.1. While it is quite likely that the working group is prepared to foreclose future minor versions with as large a scope as NFSv4.1, it is not clear that the working group is prepared to decide that only changes like those in NFSv4.2 be made in all future minor versions. As work proceeds on , the working group will have to decide which, if any of the above sorts of changes to allow in minor versions, even if most minor versions do not use these additional forms of change. As part of that, the working group will have to decide: Whether to impose restrictions on use of certain of these additional forms of change. Such restrictions could include periods of experimental use or obsolescence windows. They might also limit the combination of otherwise valid sorts of changes. For example, some changes might be allowed to be required at initial introduction and non-XDR-based changes might be allowed as well but required non-XDR-based changes might be excluded. To determine how these additional kinds of change can be integrated in the documentation model within , which is now focused on non-required features that only consist of XDR-based extensions.

One way of dealing with the issues surrounding these rules is to re-organize them so as to move away from a single set of minor versioning rules, while adapting them to the evolving versioning model in . Although the result may not have the format of a numbered list of rules, there will be sets of guidelines that serve the function. One possible organization is the following: Rules defining the types of protocol changes that may be made. These would include the XDR extensions mentioned in the existing rules, but there would also have to be provision for at least some of the non-XDR-based changes that have been useful in the past. Rules governing the construction and organization of features. Rules governing the incorporation of features in the protocol, whether this as part of a published minor version, an extension to a published minor version, or as a correction to a published minor version. Rules governing the changes of feature statuses in a minor version. Such rules would also deal with possibility of post facto changes in the composition of features or of inter-feature compatibility constraints. In this same category would be handling of feature re-organization including split and merger. Rules that deal with the interaction of multiple minor versions, on the model of rules 11 and 13 in . These are the only rules directed to implementers, as opposed to those creating minor versions. Depending on the decisions the working group makes, some of the above rule categories may not exist.

Since no substantive protocol changes are proposed here, no security considerations apply. As features and minor versions designed and specified in standards-track documents, their security issues will be addressed and each RFC candidate will receive the appropriate security review from the NFSv4 working group and IESG.

The current document does not require any actions by IANA. Depending on decisions that the working group makes about how to address the issues raised in this document, future documents may require actions by IANA.

&RFC2119; &RFC0793; &RFC1094; &RFC1813; &RFC2780; &RFC3010; &RFC3530; Primary Data Dell Work in progress. Primary Data Dell Work in progress. Primary Data Dell Work in progress. &RFC5661; &RFC5662; Primary Data Work in progress. Primary Data Work in progress.

The author wishes to thank Chuck Lever of Oracle for his helpful document review and many important suggestions.