To implement SMB2 unix extensions, smbd needs to implement ntfs reparse points to present symlinks, sockets and other special files to clients. This talk will present an overview of what reparse points are at their core and where Samba stands to implement them. Also, it will highlight the current status of the Linux kernel SMB client implementation regarding reparse points and special files over SMB2.
Learning Objectives
SMB reparse points
SMB2 unix extensions current status
Presenting unix special files like sockets and fifos to linux clients
Presented by Volker Lendecke, SerNet GmbH
Learn More:
SNIA Website: https://snia.org/
SDC Website: https://storagedeveloper.org/
SNIA Educational Library: https://snia.org/library
Twitter: https://twitter.com/
SNIA LinkedIn: https://linkedin.com/company/snia/
Foreign To the third day of the storage develop conference um today we have an SMB track and I’m gonna do the start on SMB I’m going to talk about repass points to me they are they have been new for like the last 12 months or so I learned a lot and I will
Tell you a bit about what we are going to do with them and where we are at this point my name is I am long term I’m more long-term member of the summer team I work for Zenith in göttingen in Germany we have many customers asking us for development tasks
But we also have customers that are actually running Samba and just this morning like at 4am when I woke up a colleague contacted me hey it would be great if we could have SMB Unix extensions so that our customer XYZ doesn’t convert back to NFS for it for his Unix lines and
So all this will be described in the next slide we are going to do smb3 posix extensions and what I want to do is make SMB a competitor to NFS and if you listen to Steve’s talk about this topic he always says this is more than just posix
Much more than posix he speaks about hundreds and hundreds and hundreds of syscalls many hundreds of them being related to the virtual file system so for us it’s always I mean people have been talking about this for years um it’s a question when do we claim victory that posix extensions are done and
I think we can never really claim victory because even NFS isn’t fully posix I mean try to unlink a file that is still open on NFS and you will see weird things that won’t happen on local X4 or something what this little chat with my colleague
This morning told me we have a customer with very specific requirements first they need some links to work and they need to be able to do a mod from a Linux client in posix compatible way and so these are much more concrete and much more um approachable goals that we say okay
We have individual aspects of the protocol that we need to make work and there are specifications for all of this so what what do we need to do to make SMB competitor to NFS SMB has is a file file pro file sharing protocol you can open files you can list them you
Can delete them and so on but in detail there are different semantics for example okay sensitivity some weird fellow names are not allowed on SMB that would be allowed on NFS or in posix and so on so we need to do something to extend the semantics of SMB
And thanks to the negotiate contexts that smb2 or rather smb3 has given us have given us we can now extend the protocol in a compatible Manner and what we did is we invented a new negotiate context essentially asking the server in a very early stage hey are you able to do posix
Extensions and if so um I would be as a client I would be asking for positive semantics on individual calls this is what I call here the new negotiate context then we will have to define semantics for this file name that I spoke about um yeah we need to be case sensitive
This is basically done this is also already existing in the old smb1 posix extensions that hopefully nobody uses anymore and to implement this on a per call basis we also invented a new create context you can just have extended attributes essentially on a create core extending it with arbitrary data and we
Invented the posix create context um using which you can tell the server for this particular call I want that posix semantics then although you have a ton of different information levels giving you all sorts of NTFS information from any file on the system we still have some things that
Are not really covered well so we had to implement um some new file information classes for example in links in the stat information isn’t really well covered in the SMB standard information classes we want Direct Control of permissions ownership and all of structs that
Um I mean we want to do as much as humanly possible inside normal standard SMB protocols um but there are some things we need to extend the SMB protocol with like proper ownership or something this talk is not about any of this as you can see interested in the posix specifications
David has put together a very nice document where at least the format should look familiar to you it’s this year and anybody who has looked at SMB specifications it’s just from the outside it looks like it’s an SMB extension and this is exactly what it’s meant to be
You can see this on codeberg.org David thank you great source of information I mean it’s not long because we do as much as possible inside the normal SMB specs but here you can see we have new content a new create context we have a new um create context we have a new negotiate
Context we have in your career context and so on we have new information classes so anybody interested in this um can take a look at this why a shark knows or uh Wireshark doesn’t know about these already so it’s all good that if you want to implement
That you will be able to analyze your traces properly what will I be talking about here about something that existed in SMB since when did they invent three pass points Tom when will we pass points I mean even in smb1 you have three pass points and the ability to query them right yeah
Yeah I told you I said repass points have been there forever yeah the comment was that they are a file system object essentially that is expressed over some I mean you do have some Flags in the protocol I will show them SMB definitely supports them yeah correct but
The point is they have been there forever we could have done this in 1990 whatever in the 1990s or I mean maybe early 2000s we haven’t yet but now we are approaching to utilize re-pass points properly to have extended file type support I mean one of
The requests I got this morning was I need some links and Windows NT actually or I don’t know if NT does it NTFS actually implements Sim links and they are properly exported now over SMB with a proper error message and so on but there’s more NTFS doesn’t know about block character and fireforce
And so on I mean you do have name pipes but they are very completely different object inside windows um we have to represent them to make a proper home directory server for Gnome and and so on what we do do well is regular files and directories I mean that’s the whole
Point of a file server you want regular files and directories but how do we represent the other ones I mean where is Samba sitting Samba is sitting on a Unix file system one way or the other um and Samba does find special files like character block devices sockets fireforce and so
On on the file system how does samba how does somebody deal with them right now I can only say badly so for years we have had the ability that a server can create a fifo on the side on that that a demon on a sambasha Sheikh can create a fifo
And that a client connects to it and writes and reads we broke that nobody noticed we broke that because some when you when you talk to a fifo the Linux system will only accept pure read and write calls it will not accept read V give you know an offset directly atomically and rightly
It will give some weird error message and we removed the normal read and write which is the right thing to do but it broke five volts nobody noticed we don’t have a back report for them and I mean like five years ago we don’t do since
Five years ago we don’t do read and write anymore sockets different thing I mean you could create a socket on the Unix file system and expect some level of semantics over the SMB protocol that you can connect that you can read dating when right we don’t do any of that
What we do right now is a demon on Linux can create a socket and over the IPC dollar share you can connect with the named pipe calls that are there so you can utilize them but only in a very very specific Manner and we use that for somebody’s erpc or RPC system
Block and character devices I mean what do you want to do if you have Dev SDA in your share the character device the block device do you want to really want to export that disk over SMB I mean I know there’s essentially tunneling of iSCSI inside SMB but this is not what I
Would expect a Unix server to do so what we want is we want to present all these special files in a way that a client can use them locally essentially the this list board entry pass points according to Wikipedia a re-pass point re-pass points provide a way to extend
The ndfs file system a repos Point contains a tag and data that is interpreted by a file system filter what the heck does this mean so what I found out by testing and and playing around with them over SMB is my interpretation of things I I don’t have
Access to Windows Source but essentially re-pass points are completely normal fast you can read and write to them if open properly you can they even they even respect share modes I have tests for this the only thing that happens is when you normally open a re-pass point like you would open an application
Very very late in the open call one of these filter drivers steps in and sees hey there is an extended attribute on the repos point there is a tag and when it is there then very late in the process of opening a file we go through the list of filter drivers that
We have does anybody know about these repos points and then call the Callback and do something and if it’s not there get a special error message if status I all repass tag not handled and that’s the only thing I could find when re-pass Point what what repast
Points actually do so when opening a file very late in the open process if I don’t open a re-pass point as such as a repass point if I do a normal open somebody can step in with a callback and modify that file on the Fly so
Msfscc defines a few dozen of these tags that can be set as extended attributes essentially on NTFS files most of them are labeled as not meaningful Over The Wire the specification says two types of free pass tags are documented actually that’s a little bug the smb2 document or fscc document
Specifies three of them and we will use them essentially when they when you read not meaningful over the wire is we don’t think we need to document them but I mean the ones that we are going to utilize are documented um as a norm file or opening it as a
Re-pass port yes yes that’s a normal Cricket request open that the question was I spoke about opening as in a file normally and as a repos point the only difference is this is a normal call to open fifo and the difference between normally opening it opening it normally
And as a repass point is this bit here so at the career in the moment of the create call the client can tell the server hey I’m interested in the normal file in normal semantics and whatever please reinterpret this re-pass point for me do whatever you want with it
Like whatever open there’s there’s definitions for HSM which means that a hierarchical storage manager could step in when you open the file um and just pull back stuff from tape before returning to you let’s if this bit is zero which it is here then um all this mechanics step in so but
This Firefall is not handled here in that file server so it is a file with a repast tag which we don’t have a filter driver for and what you see here is an error message and what we do next is we create again same request all bits
The same it except for this one open repos point and this tells the server hey don’t walk the list of filter drivers to see if anybody has any business than this just give me the raw data without interpreting and this succeeds and the point that I wanted to make
Earlier is when you get when you have this handle you can just normally read write truncate whatever you want with that you you have share modes I mean you get share modes for these open chords even if you say okay I have a re-pass point I don’t want the
Filter drivers to step in everything just happens as it was worth normal files it’s a normal file with a special attribute we’ll get to that any other questions good so where do we come from we want to represent all these special files and we see that we have a few dozen repast
Tags defined in msfsac and there’s two special Definitions of free pass points for all these two objects where do they come from Microsoft implemented the windows subsystem version one version two works completely differently but they hit the same problem they wanted to present Linux semantics based on Windows NTFS and what they did is they invented re-pass tags
Just files with extended attributes they have a name they have all attributes like you would expect from a file but they invented repast tags for example we have re-pass tag AF Unix is used for socket and so on the problem is none of these are documented and they are definitely not used anymore
At all because wsl1 is essentially completely deprecated I don’t know if you can still install it if you want to WSL version two essentially is a completely complete Linux kernel using X4 on a block device so they don’t have to mess with any special semantics being added to NTFS files and then
There is the windows NFS server and that also has the same problem the windows NFS server um needs to cope with a client coming in and saying okay I want a fifo I want a socket a water block device and they invented a completely different set of
Re-pass points and the point is these are actually documented if you look at msf sec oh excuse me these are actually documented in msfscc and what these guys did is they didn’t talk to these guys and look at what they did but they invented one more repast tag and they added um
A sub definition of or a sub tag am I a socket Mia block device or Mia an NFS some link inside the blob that is covered behind ioriba stack NFS if you look at the specification the obvious thing happens for block and character devices you need major and minor numbers
And they are just so you have a repast tag IO repastag NFS with a sub tag telling I’m a block device and then you have 8 bytes four bytes four major four bits for minor for Block devices as the data content of this repast tag attribute simlink has as Target a unicode
Swimming Target essentially I mean they could have used the normal posix siblings but they didn’t and fifo and sock as special repass tags don’t need additional data they are just being given a special tag inside the i o repas tag NFS data definition you can look that up
And let’s look at what Windows does with these so here wait a second I know you can’t read that but I found this key now you should be able to read this so these are files that I created over an invest client via an fx client
This is a block device this is a fifo you can you can see what I mean the bar is a Target of assembling or it’s an invest swimming I don’t know anymore but they are represented here as completely normal files and if you go to the command line huh still another thoughts
I’m sure that at some point at some in some place I saw that wait a sec possibly if I come over SMB I get special attribute or if I look at this properties here somewhere all these re-pass points have been listed as siblings I don’t remember I I’ve seen this so um here
Live demo time always breaks so this is a proper NTFS filming this is the one that is actually being used in the real world I mean who uses the windows NFS server it’s just there but this is actually being used and this is what explorer is aware of so what
Explorer these sees when listing this directory it sees hey this is a repast tag and all it does is it it tells you it’s assuming maybe they fixed that in some update I’ve done run updates on this machine but what I’ve seen is that Explorer also
Has all of the other ones here as Sim link so this is not really consistent to the end even in the windows world so um but what you see here is a use of normal siblings let me exit this here NTFS siblings NTFS oops so now we have to make a choice
We want to represent all these special objects what do we use and my Strongbow vote is that we use this year why WSL has distinct re-pass Point tags so we have the wsl1 definition and we have the NFS server definition I would strongly vote for the NFS repos point definition
WSL no longer used and probably they won’t reuse these tags these three pass tags for something else I’m sure but still um the format is completely undocumented I mean you will be able to figure that out um but it’s it’s completely undocumented um NFS only uses one repos tag
And if you look at what speaks what what is arguments Pro NFS we do have documentation and what we can do is we can do protocol level tests against the definition of interoperability which is the windows NFS server on NTFS also being exported by SMB
So we can play with hey this is these standard for this kind of interoperability let’s do the NFS make link NFS whatever NFS make socket make fifo and see what happens on the SMB side and you want to be compatible with that WSL saves us a round trip on directory
Listing because it has distinct re-pass tags um per object so when you do directory listing in the query directory call you only see hey this is a repass tag for NFS it could be a file for it could be assembling it could be a socket and to satisfy LS minus l
To print as a p or a s or a whatever we need to ask we need to look in the data contents of the NFS Rebus tag and that’s yeah that’s another round trip but I think we have to buy that bullet big trouble makers we had our Shelf security issues with
That but they are there they are not going to go away although Jeremy really wants them to go away but they are going to stay at least until I retire I’m pretty sure so for some minutes we have even three options there is the WSL specification there is the NFS specification
And as I mentioned quite a few times already here NTFS has re-pass tax Zoom link and this is one of the moments where I where a standard Windows Server can actually do something with a repos tag I told you that when opening a file very late in that process all the filter
Drivers step in and say okay let’s do something with this and the repast tag swim link is being properly dealt with in a standard NTFS in a standard Windows SMB server and what does it do if you try to cross a Sim link when opening a file you get a special error message
As we have seen already when opening assembling with a Firefall that nobody can interpret you get I already passed tag not handled no filter I mean this is the re-pass tag but we don’t have a filter drive over it every NTFS does have a filter driver apparently for Io repast tag swim link
And does something with it what does it do it gives you a special error message there you go so this happens when you CD into a directory or whatever I don’t know I think I did a all info or get on lnk which is an NTFS Sim link on a Windows server
That you can create with MK link on the Windows server and what it does it returns hey stopped on Zoom link and what it also does it does is it gives you a very informative error message namely ah let’s open this crap namely what it does is
It tells you hey the zoom link is at this point of the path you have given me so imagine your open a slash b c d slash x dot text and somewhere in the middle for example B points somewhere else Windows is always completely path based
So you always give it the absolute part from the root of the share to open a file and if somewhere in the middle there’s a sub link in smb1 what you would have to do you also get that error message over smb1 but what in smb1 you would have to do is
Just open the individual components piece by piece and ask the server hey is this assembling or not and where do you want me to go smb2 error message gives you this unpassed path length so it tells you those bytes those 10 bytes of the path
You give me I have not looked at this means that slash B is the same link and where does it point the symboling points here and the interesting thing is that this can be anything depending on Windows client settings this can even be backslash backslash server name backslash share backslash path
So essentially what we have done for msdfs for example is we have a special error code path.covered and so on and then the windows line comes back to us and where do we need to go you could all do essentially all of this with some links
So what you can put here is the signaling Target and this makes it easy for the client to interpret all this stuff on the client side which is good um the question was what happens if there are several repos tags in the path my guess would be I haven’t tested this
But my guess would be that um Windows just goes from the top it does the path to reversal and whenever it hits the first Zoom link it just bails out and tells you hey I can’t I I can’t Traverse this one it’s the same link interpret that yourself
And get me back get get back to me with a new complete path and so on that that’s my idea yeah that’s definitely what will happen when windows and cameras point but I don’t think it allows more than one yeah that’s true I think I mean you can
Over it’s just one external file it’s like it’s almost like a type okay but I think the question was you have slash a slash B slash C if both slash B and slash C are some kind of repass tag Yeah exactly so my idea is that Jeremy what is that eel detection strategy client based the question was what is e-loop detection strategy it’s like an NFS NFS doesn’t deal with with some links you have to do that in the client was the question was do I know what the
Limit of e-loop detection is no clue I know there’s Luke detection in the file system but if you were to assemble a weird well yeah no no it um Tom said it’s it’s uh there is some Loop detection but it must also be on the server side
Because it must be in ndfs somewhere or somewhere because because you can also do Sim links purely server side without SMB yeah protection already isn’t it but you’re right that I mean especially true yes Jeremy’s coin was I hate some links watch his talk from from zanda XP response foreign
I didn’t fully get the comment but I think Windows just stops at the first encountering of a review pass point it wouldn’t even it would I mean it wouldn’t even look further and and it just would treat I guess the rest of the path that you have given to Windows beyond
That re-pass point as gibberish and Klein deal with it and passed exactly that’s what it says I I start on this component I ignored all this so it’s completely up to the client and as I said you can send the client to a different server with this with proper
Settings on the client it’s some FSU tool settings that you can say local to local local to remote remote to local and so on um activate it or not but you can send Windows clients to other servers if properly configured on the client so what do we do
In posix extensions if somebody coming in from the windows client wants us to create assembling the big thing is no we won’t create a server level assembling what we will do is or how would we create siblings over SMB there is a standard way for it essentially just create a repass point
And the client will use that facility create a file with repass tag fnctl set repulse point and be done with it and we will just follow what the client tells us create a file set that with a special extended attribute set the attribute yes I’m a re-pass point and be done with that
The option would have been that somewhere in these two steps we actually turn this into a real server-side swim link and we won’t this is what gives us security headaches all over the place and moreover you can’t do this because these two steps I point out just
Create a file with openre pass point and I think you don’t even need that you create a file and only later the client tells you this will become a repass point and there is on Unix there is no Atomic way to turn a file into repass point
So when we have created that file as a normal text file or whatever it’s too late so there is no Atomic way to actually turn this into repos point which benefits us that we don’t even start to mess with re-pass points on the server side with this assuming is being created over SMB
Jeremy does Windows have a restriction if you open up our right data into it and then turn that file containing data into a reform the question was does Windows have a restriction that you can turn a non-zero or a file with real contents into a repos point
No it doesn’t you can have a 10 megawatt file turn it into a Sim link essentially there are restrictions for directories you can only turn a server-side created directory into a Sim link when it’s empty that’s what it is and if there is a re-pass tag on a directory I node on the
Server side you can’t create files in there anymore but for this I would strongly vote for we don’t do that on directories we just don’t because it causes us all all sorts of other trouble with our internal optimizations and so on this just won’t work so windows in Windows you can create a
Directory turn it into assembling we will not support this live so again um the question was can the assembling Target that we create as a repast point on the server side we a directory it can be anything it’s a Blog it’s a Blog that the client has to deal with
We just return the blob period and the blob is essentially this whole thing here repass data length um so this is the error message and you have essentially it’s this object object these blue bytes there that we have to store and that we have to do get into
The fs control set repast tag with all the data in there unpassed path lengths it’s a reserved field you have flags flags meaning Flags one meaning it’s a relative or absolute sibling and then you have these two Fields first the substitute and the pretty printed name and so on when
Creating a symboling over SMB you have to give it this whole thing to a father for the server-side filter driver to turn this into an error message stop on something but but beyond that it’s completely a blob I mean I don’t I haven’t really checked if we tell it repast tag
Swimming and then just give it like a 10 megabyte random set whatever whether they are any consistency checks sorry Steve was first Google that’s curious and it I haven’t tried it but it said that 16k maximum for uh for the reports data so we say the the comment was that there is a
Maximum size for repos data might be true but in general I have already tested that you can in general just send garbage as part of this there might be size limitations it’s less likely that you’re going to put an enormous virus in something if this is true the definition that says
That it’s a capital size don’t know yeah Jeremiah I actually took today And he confirmed that in NTFS it’s stored as an extended attribute so all the limits on size of extended packages yeah the common was that essentially even it’s implemented as a normal attribute with some special whatever magics icing on top but it is extended attribute with all the restrictions that
Apply to Extended attributes oops okay so where are we now we have a server code I mean not in master we have server code that where you can set and get re-pass tags on normal files we also do have server code that gives you for real siblings and we
Tested this in the io lab already for real swim links also we export a share there is a Unix level swim link and Samba can now get the proper stop on Sim links including the metadata redirecting the client and this is activated right now we have this option
Follow Simplex equals yes or no so you can make some or follow assembling server side some would say bad idea from the start but it’s there and people depend on this right now the code I have presented to Steve is that this step stopped on something like happens if you if you say
Stop on simply follow siblings equals no and if you then come in with posix open request so you get a create request of the posix create context it will always activate follow something first now so we will not have in the posix world in the posix extension world we will not have server
Side followed siblings at all um I mean then next question apparently fifas are broken shall we use those as a test bed for this logic with getting fifos same for sockets and block and character devices we could just say if a client comes in and does a create on a non-normal file
We give it re-pass Tech not handled and then the client needs to figure out hey get me what it is we could do that pretty simply the question is does that break user scenarios can we just activate this do we want to activate this I mean I have code for
Most of this already do we want to activate this by default just do it Jeremy well there’s nobody noticed that binders were broken I would say Jeremy’s comment is that we don’t have web reports for five was being broken just do it Steve the comment was that in the return of the
SMB create we have a Flags field and we could activate hey this is a re-pass point um but I mean what do you do with that handle then I I would vote for I would vote for repost Tech not handled and and just not deal with it and yeah Steve
We have code today to um the thing I was curious about is should we allow it to me I don’t have code to create ficos this way I mean obviously the older ways we do um but we don’t want to be using the you know 2080 year old way with extended
Actors you do this we’d rather be doing it this way um if you want tomorrow or today we can we can do the create um yeah the comment was that um the Tiff’s client right now has code for creating a fifo using the repulse tag method we did you don’t ask
Query okay and recognize that so we tested that to Windows um but creating the photos and creating block and creating care devices we only do the old style bike but creating it you can do that today or tomorrow if you want yeah so um the comment essentially was for the for the recording
Um that right now is no code to create fifos and special files using the simlink method that is laid out here and so but that should be relatively simple to add I guess yeah um what is incomplete is doing all of that over smb1 but the question is [Laughter]
And uh so it’s not complete but we need to do the question is do we need to do that probably no um and right now I’m working on lip SME client to include user space clients and there’s one little demo that I could try to give you here and that’s this
Call essentially what do I have it is a normal SMB server localhost and what we have in Samba is our usage of is examples of our exported lip SMA client that use this weird SMB URL and that can then issue a stat call and I have extended this such that and let
Me open a Wireshark Trace TCP dump term stat cap so this is an example client of this exported lip SMB client and the point that I want to make here is this that is that this code is now able to fill in this stat field with the
Contents of the repast tag and if you look at this is the exported share uh I think it’s git share Firefall this is the object server side oops this is the object server side is it is a 50 and we correctly now report this as a fifo and if we now look at
The Wireshark trays currently being taken in term so this is live let’s limit to SMB to what it does it tries to and now you should be able to read this I mean this is the sequence that you have already seen but it’s live um I’m done almost
Um fifo we get a repos tag not handled then we open and just trust me with repast open as re-pass point and then we do the get in for everything and then we do the get repos point on the fifo asking hey what do you have and this piece here then is artificially
Generated as a repass tag NFS and it’s a fifo and yeah so this works other side client support is still a bit lacking in particular with siblings and so on I mean that’s that’s a nightmare on our in our client in the in the users Baseline SME client
Lip SME client and so on that’s that’s a nightmare to get right in all cases okay and with that thanks for your attention it was a good discussion thank you [Applause] any questions beyond what has been asked during the clock we have three minutes left yeah
For a client we tries to create a symbolism over SMB that actually points that a global assembling the points the different SMB server the question was will we accept we at some links pointing outside server and so on um yes we will because essentially what I said it
What we get is this blob and essentially we don’t really care about this blob it’s a blob that we store as an extended attribute we can always say okay do some sanity checks and look at the contents of this block we’re at points and whatever forbid pointing out of the
Share or something or only support relative or relative and non-absolute summonings and so on we can do all we can apply all sorts of consistency checks but server side we don’t care it’s for for the server it’s not a security issue because it’s just a blob that we saw in
An extended attribute right you said that this is only working when follow on Sim link is disabled correct what happens if you encounter an actual logic the question was what happens if we encounter a positive swim link We artificially generate this and the only thing we take from the file system is
The substitute name and we will have to apply some magic and if code for that we look at the right now we look at the code at the assuming Target and make a check does this point into the share if it’s an app if it’s a
Relative Sim link we just report this if it’s an absolute server-side swimming we the the current code says does it Point inside the share then we cut off the share route make that a relative symbol but that’s all that’s all magic that can be discussed so if we have an absolute share pointing
Back in the same directory server side we turn it into a relative assuming that then the client will interpret properly hopefully if none of these heuristics make sense we just pass it back as is Steve obviously today special files we can use SFU which should be support X adders or not we know
Or not um if you you support and then simply it’s almost work right we do support and have some links today but if we’re going to do this for let’s say 5000 or whatever else one of the problems I say is how do we know um like a normal server let’s say windows
You can either create a request point or you can if the file system supports creative reports points all request Point creates will work if not they’ll all fail here what we’re looking at I think is that we may get errors creating a typo with the NFS file but never but no um
The question was um when ndfs supports creating re-pass points or re-pass points we’ve created will be the same here because for us we will implement this or we I have code and based on Jeremy’s code that implements repos point just as an extra build and we don’t care and this
Is exactly his question we don’t care what what the contents of the executor is we could apply all sorts of magic and send it to checks and so on some kind of kind of filter but the server doesn’t care I’ll read those I’ll see them however I want and you don’t have to worry about them correct and I think with that we are one minute over hey out of time perfect thank you thank you [Applause]