### A powerful file server for Windows ** _ No part - reflections without memories. _ ** Without further ado, let's start by thinking about what a file server is, how it functions, and what we need to make a file server as fast as possible, for example, to make the most of our gigabit network. At first glance, everything is simple. The client came, asked for the file, climbed to the disk for the file and gave it to the client. Much easier. On the second and more advanced views - everything is much more complicated. A file server is more than just a network interface to a hard drive. Of course, the reading speed of modern hard drives is not low, but what will happen when not one client arrives, but several dozen? In addition to having to share this very reading between clients, you will also have to jump, reading data from different places on the disk. And the access time value is already quite decent: Mechanics, pancake, no performance. In order for everything to work quickly, and for clients to have the same impression from working with a file server as from working with a local disk, this mechanics must somehow be replaced with electronics. Those. clients should receive data not from the hard disk, but from RAM, which should be enough. The standard way a file server works is Disk -> memory -> net Of course, the processor monitors all this. So, let's take as a starting point what we should: 1. Work a lot with memory, moving data from the hard disk to memory and from memory to the network adapter. 2. Do it all at the same time. There are two and a half ways to make the transfer of any data faster - increase the transfer rate (for example, the bus frequency) or parallelize the transfer. The remaining very effective half-way is to avoid transmitting data altogether wherever possible. Most of all, the work of a file server resembles the work of a cardiovascular system with two circles of blood circulation. And, as we will see later, it is difficult to save on it. **_Part one - choose a platform._** **64 bit boom?** The area of RAM that stores data retrieved from disk is called the system cache in Windows. The larger the system cache, the less often we have to climb to the disk for data. We eliminate the need to transfer data from disk to memory. Half the work is done without any effort. A 32bit address space is 4GB of addressable memory. That is how much any 32-bit application can address, including the kernel of the system itself. On Windows, an application can use 2GB (3GB if the /3GB key is in boot.ini, but this will not interest us at all) for its data. An unpleasant fact follows from this - for the 32-bit version of Windows, the maximum size of the system cache is 1GB (more precisely, even somewhat less), which is not much by modern standards. And sticking more than 2Gb of memory into the file server will not only not help, but will also aggravate the situation, due to less efficient use of the processor cache. All this is eliminated by using 64-bit platforms. The 64 bit version of Windows 2003 supports a system cache up to 1TB. Thus, a file server is exactly the application in which the use of a 64-bit platform will not be superfluous if we want to achieve efficient caching. **Chip and Set to the rescue.** Let's now admire why you can't use desktop chipsets in a good file server, even if they are very good and productive. Let's look at the example of two chipsets on opposite sides of the border between a server and a workstation from Intel. Other manufacturers do things similarly, but Intel makes it much easier to find the right pictures. Let's compare the highest performing desktop chipset 955X (Intel puts such chipsets in a separate category between desktops and servers, which it calls "Workstation") and the minimum entry-level server chipset E7230. The characteristics of the chipsets are absolutely identical - memory frequencies, bus bandwidths, supported processors and the southbridge circuit. The differences are visible if you look at the chipset diagrams: ![Intel 995X](Intel995chipset.gif) rice. 1 is a diagram of the Intel 995X High Performance Workstation chipset. ![Intel 7230](intele7230chipset.gif) rice. 2 Intel E7230 Entry Server Chipset Diagram The question is, where are we going to throw the dice, in the sense of sticking equipment. The 955x uses a fast 8 Gigabit PCI Express x16 bus for graphics. To connect disk and network adapters, we can use PCI or PCI-X via the southbridge (ICH7R). But the performance of the DMI bus between the southbridge and northbridge is 2GB/s, which will be shared between the disk and the network. The bus is synchronous and here are gigabytes, not gigabits, i.e. the speed is quite decent, but still ... The server platform has an additional PCI Hub that provides connection to PCI-X ports via the PCI Express x8 bus and PCI Express x8 and x4 ports, allowing you to fully use even 10 Gigabit Ethernet, and with all this the disk subsystem will be able to use another bus. The "server" platform, the "northern" are the high-speed peripheral buses and the more of these buses are connected directly to the north bridge, allowing you to move data from external devices to the memory and processor with minimal delays. **Memory** With memory, everything is clear. This resource is the most critical for us, so the memory should be as fast as possible. **CPU? What processor?** On the one hand, the processor in the file server always remains a little on the side of the main data streams, if the network adapter supports hardware checksum calculation and data segmentation, and unless, of course, we foolishly decided, for example, to encrypt all traffic. On the other hand, it is not superfluous, because each packet that we send to the network must be formed: And this packet is not large, which means there will be many of them, especially if the network, God forbid, is very productive, and there is work for the processor . In addition, our processor manages and receives messages from various devices - network adapters, disk controllers. The sooner he processes these messages, the better. Even better, if messages from different devices are processed in parallel on different processors. So, when choosing a processor, you should consider: 1. Strange as it may sound to connoisseurs of parallel computing, two slow processors are better than one fast one, and the more different controllers and adapters we have, the more advantages we get from multiprocessing. 2. The more parallel memory accesses (the larger the system cache), the stronger the influence of the processor cache size. Moreover, the caches of two different processors are not summed up. 3. A lot of powerful processors will be needed when we get to a multi-gigabit network. 4. The processor bus frequency can affect the memory bus frequency. Therefore, you need to choose a processor that works with a bus frequency that does not slow down the work with memory. **Network subsystem.** As we have already calculated, 1 Gb / s (Gigabit per second) and 1GB / s (Gigabyte per second) are not the same thing at all. If we look at gigabit Ethernet, then the real transfer rate for a more or less long period of time in it will be no more than 90 MB / s on one port due to the asynchrony of the bus. And to serve a large number of clients, gigabit Ethernet may not be enough. The first alternative is to upgrade to 10 gigabit ethernet, which is expensive and not very standard. This is especially true for adapters that can work with high-speed versions of PCI-X or PCI Express x4/x8. A solution like this can pay off on enterprise-grade servers. As a cheaper and slower alternative, you can have many gigabit adapters. Drivers for many adapters allow you to combine them into one virtual one to improve performance. But remember that this mode of operation may cause bewilderment for some switches (switches) and necessarily requires careful testing. The performance of the PCI bus is not enough even for a gigabit adapter, taking into account duplex transmission, it is better, of course, to use PCI Express or PCI-X adapters. Dual adapters are not recommended. For 10 Gb/s Ethernet, you must use PCI-X 266/533 or PCI Express x4/x8 ports. PCI-X is a synchronous bus and its real performance is close to the maximum 1 GB/s for PCI-X, 2 GB/s for PCI-X 266 and 4 GB/s for PCI-X 533. PCI Express - asynchronous bus and real speed transmission below the theoretical maximum. But ideologically, PCI Express is closer to Ethernet. Most modern adapters support "TCP Offload" capabilities - ie. transfer of computational tasks associated with the organization of a network connection from the processor to the network card. Usually, at least two tasks are supported: packet checksum calculation (Checksum Offload) and large packet segmentation (TCP Segmentation offload, in some Realtek and HP drivers this is called TCP Large Send Offload). For a productive server at gigabit speeds, these tasks are critical, because. account for up to 30% of all server computing tasks due to the fact that the entire data stream will go to the processor to calculate checksums. Therefore, it is very important that the network adapter perform these tasks, and perform them correctly. In the case when traffic encryption is required, the network adapter must also support IPSec Offload. Adapters with IPSec support are significantly more expensive, but even on them, performance of several gigabits will remain a dream. **Disk subsystem** The first question is SCSI or SATA. If we are not chasing hundreds of megabytes per second, then SATA will do. For high-performance systems, of course, still SCSI. The disadvantage of solutions on the SCSI bus is small volumes and high cost per volume unit. But she's faster. 320 MB/s in Ultra320 SCSI is 10% faster than 3 Gb/s in SATA II, again because of the tricks of the serial bus, which is SATA. The real stable read/write speed of advanced models of SCSI drives is also about one and a half times higher than SATA-II drives, approaching 100 MB/s. Another advantage of SCSI drives is that they are much easier to pack in large numbers. The more disks we fill, the higher the performance of the disk subsystem will be. disks can be accessed in parallel. Of course, having practically SATA at your disposal, it’s a sin not to use it at least somehow, for example, to place a system or even part of the data on it. Based on the above figures, we can strongly recommend using two-channel or four-channel SCSI controllers and do not hang more than 2-3 disks per SCSI channel. If you need to have more than 6 drives, you need to have a quad controller, use multiple SCSI controllers, or look towards faster interfaces such as Fiber Channel. Today, there are already a sufficient number of high-performance SCSI controllers, for example, MegaRAID from LSI with support for PCI Express up to x8. We'll be fine with PCI-X or PCI Express x4 for a well-packed dual-channel controller, but regular PCI Express won't be enough. Ideologically, the SCSI adapter is closer to PCI-X. Second question: which RAID to use? The question of whether to use RAID for file servers is not worth it. In this case, there are two tasks for RAID - to ensure disk fault tolerance and to maximize the speed of the disk subsystem by parallelizing writing and reading across disks. Of course, the fastest RAID \ is RAID 0. However, it does not have fault tolerance. RAID 5, unfortunately, slows down the write operation, especially when writing small pieces of data, but it remains quite fast when reading. This is a consequence of the need to recalculate and rewrite the checksums during the write of an incomplete block. RAID 10 (one-zero) has a well-balanced read and write performance, but we lose half the capacity of the disks. Therefore, we can recommend RAID 10 for storing data that is constantly being processed, such as current documents, RAID 5 for various archives and storages, and RAID 0 for temporary data, such as an optimizing proxy cache. Naturally, for better parallelization, it is better to use the maximum number of disks in RAID, preferably from different channels. **Frame ** Well, what do you think, what kind of case you need to have in order to put all this into it: * fit * Powered * Not melted ?? _From what was said above, two types of file servers are immediately drawn. The first is an entry-level file server with performance up to gigabit per second, for which there are practically no requirements for hardware (provided that it is modern enough) - in fact, you can use good desktop hardware with integrated SATA RAID 1-0. In a multigigabit file server, you won’t be able to save on anything, it will cost orders of magnitude more expensive._ **_Part two - choose and tune the system._** ### Choose a system With the choice, everything is simple, you need Windows Server 2003, for a 64bit platform - 64bit edition. We will not have special advantages from the Enterprise Edition if clustering is not required. We do not need a large number of processors, in the case of the 32-bit version, we still will not stick more than 2GB of memory, and in the case of the 64-bit version, memory limits are not critical. _It is better to use the English version of the system on servers \- in case of problems, it is easier to find the corresponding article in the knowledge base __Microsoft._ ### Setting up the system Actually, the standard system setup for a file server is minimal. In the properties of the network connection, you need to open the properties of File and Printer Sharing for Microsoft Networks (file and printer access services) and put a bullet on Maximize data throughput for file sharing. ![Windows Optimization for File Server](FileSharingOptimization.jpg) Fig.3 Microsoft File Sharing optimization selection _There was a bug in Windows NT, the values of Maximize data throughput for file sharing and Maximize data throughput for network applications were inverted, but in Windows 2003 this shortcoming is fixed, but you can often find outdated, and therefore incorrect advice to swap them._ We create the necessary partitions on hard drives, of course, using the NTFS file system. Set the necessary access permissions (ACL). The ACL should be as short as possible. The correct list contains, as a rule, 1-2 entries, for ordinary users and administrators. The ACL is checked every time the file is opened, the shorter it is, the less time it takes to check. Microsoft recommends the following algorithm: user accounts are grouped into small global (Domain Global) groups, usually in accordance with functional responsibilities. To access resources, domain local (Domain Local) groups are created, which include domain global groups of those users who need access. This minimizes the size of both ACLs and SIDs (Security Identifiers) in a user's access token. We open file sharing, set up various auxiliary tasks, such as backup and defragmentation. The server is ready to go. ### How does and system cache work? When the file and printer access service ( LanmanServer ) is started, a thread pool and a work queue are created for each of the available processors. Each thread a job from the queue, processes it, and is taken over by the next job. What happens if a client tries to open a file over the network? The redirector service passes the request to the client service for Microsoft networks (LanmanWorkstation), which forms an SMB request and sends it to the file server, the LanmanServer service, where it enters one of the queues in the form of a (work item). When the queue reaches it, the freed thread takes the job from the queue and executes the request, accessing the requested file fragment. When accessing a file, the MDL (Memory Descriptor List) interface is used. If the file fragment is not yet in the system cache, then it must be obtained from disk. In this case, while working with the file, the is placed in a separate blocking (blocking) queue until the file operation is completed and the worker thread starts processing the next task from the queue. The MDL interface returns a list of real (physical) memory pages from the system cache that contains the requested data. If the job was in the blocking queue, it is transferred to the main queue and waits for processing to continue. Then the worker thread generates service data for the response SMB packet and transfers this data, as well as the received real address from the system cache, to the network adapter. The network adapter obtains data directly from the system cache using direct memory access (DMA) and sends it to the client along with a service header. Thus, access to data in memory occurs only once - by the network adapter. The processor does not process this data. If the client writes any data, it is also changed in the cache and flushed to disk when it goes idle or as needed. By default, the system cache size is only 8 MB (this is for Windows NT and Windows 2000, unfortunately there is no documentation for Windows 2003 yet). Setting the Maximize data throughput for file sharing option (Figure 3) actually changes two registry keys. The Size key in HKLM\CurrentControlSet\Services\LanmanServer\Parameters is set to 3 (Large). This key optimizes various parameters of LanmanServer's work queues and thread pool to work with a large number of clients. The LargeSystemCache key in HKLM\CurrentControlSett\Services\SessionManager\Memory Management is set to 1 (on). At the same time, 80% of all unoccupied system memory is allocated for the system cache, which will significantly increase the performance of a dedicated file server. _I regularly hear things like . Very often the reason for such statements is that a server with a small cache is selected for testing and one client is used. All network operations on the client are also cached. The same cache size and caching algorithm leads to the same data in the client and server caches, i.e. all requests that reach the server fly past the cache. Of course, this leads to the fact that the server is quite slow. Windows 2003 will show itself to the fullest, provided that the right hardware is selected, the appropriate settings, and on a sufficiently large number of clients._ **Optimization and overclocking of the file server.** Now I'm going to disappoint everyone. The real file server optimization happens at the hardware level, which we've already discussed. The that we are going to talk about will give a gain of more than 10% only on poorly balanced equipment. Eliminating the bottleneck in the hardware (for example, replacing the processor) in such a case will give much better results. As we can see, the operation of the file server is quite simple. All we can do is reduce any delays - the time it takes to open / close a file, to simply request a queue, etc. Everything else is purely a matter of hardware. Due to the peculiarities of the cache and request queue processing, the file server will not show performance miracles if any applications are running on it that use memory intensively (especially with constant allocation and deallocation) or load the processor unevenly. You should not give the file server any additional roles in the network, except perhaps the role of a domain controller and an Active Directory server, provided that there are a very small number (on the order of tens) of server users. ### Optimization of work with equipment Microsoft recommends that in multiprocessor systems (really multiprocessor, we are not talking about hyperthreading technology), interrupts from the same network adapter are processed by the same processor. By default, an interrupt can be handled by any processor. You can use IntFiltr ([ ftp://ftp.microsoft.com/bussys/winnt/winnt-public/tools/affinity/]()) to bind interrupts of a specific device to one processor. REG_DWORD HKLM\System\CurrentControlSet\Session Manager\I/O System\CountOperations A value of 0 disables various counters for the hard drive, which can speed up I/O operations. REG_DWORD HKLM\System\CurrentControlSet\Services\miniport_adapter\Parameters\DeviceXX\NumberOfRequests A hardware RAID controller can handle more SRB requests through parallelization. It is recommended to raise this value in the range from 32 to 96. REG_DWORD HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\IoPageLockLimit How much memory (in kilobytes) the system can lock for an I/O operation. The default value is 512 KB. Increasing this setting will increase the amount of data that can be transferred in a single operation. You can find recommendations to increase this value up to 128MB, however, an increase of more than a few megabytes (for example, within 8-16) is unlikely to be effective. ### Optimize file system settings REG_DWORD HKLM\SYSTEM\CurrentControlSet\Control\Session Manager\Memory Management\PagedPoolSize Paged pool and file cache share the same area of memory. The paged pool is required to work with the paging file, which, with sufficient memory on the file server, we practically do not need. We can slightly increase the available memory for the file cache by reducing the paged pool size (specified in bytes). You can take values of the order of 128MB or even less. REG_DWORD HKLM\System\CurrentControlSet\Control\FileSystem\NtfsDisable8dot3NameCreation Disables the creation of short filename aliases. This feature is required if the disk will store data processed by MS-DOS or Windows 3.x applications (you need to keep in mind that many installers are). Disabling (value 1) slightly speeds up the procedure for creating files. REG_DWORD HKLM\System\CurrentControlSet\Control\FileSystem\Disablelastaccess When set to 1, it disables changing the access time when accessing the file for reading. The last access time is rarely used (unlike the creation and modification times). The prohibition of updating it allows you to speed up the procedure for opening a file for reading. ### Network settings optimization It's a good idea to start by disabling all unused network connections (to minimize the routing table), protocols, and clients. You can take a look here: HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TcpWindowSize HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\TCP1323Opts TCP window size. The window size should be increased in networks with high bandwidth and high latency. These can be: networks with a performance of several gigabits per second, very remote networks (via satellite or several routers), busy networks in which the number of avoidable collisions is quite high, networks with weak clients, for example, with old computers. When increasing the window (TcpWindowSize) above 65.535 (a value close to this is used for gigabit and higher interfaces), the TCP1323Opts value should be set to 1. HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\MaxHashTableSize Indicates the size of the TCP connection hash table (default is 128). The maximum value is 65535. The optimal value is no less than the number of expected concurrent connections. You can also raise it to the maximum value. HKLM\System\CurrentControlSet\Services\Tcpip\Parameters\Interfaces\interface\TcpAckFrequency Indicates how many TCP packets (default 2) ACKs are sent. Microsoft recommends increasing this value to 13 to reduce the number of packets transmitted over the network. That's all. Write. Share your impressions. Remarks. Tips.

High performance Windows file server