You can find everything about the different operations and maintenance work of a crypto mining farm in this guide. All the staff in a crypto mining farm must know about all these activities. They must master the different maintenance operations of the mining system. The different maintenance operations include miner resetting, firmware upgrade, basic maintenance, TF card recovery, judging simple faults like network failure, temperature protection, failure of fans, etc.
Along with all these maintenance activities, the staff must also have a good understanding of the power supply of the minor, the working environment, and the network. Regular inspection needs to be done on the mining system to ensure its smooth working. The maintenance staff must be able to find and fix all the problems associated with the miners. It will help to bring down the miner failure rate drastically.
We will look at the standard operation and the different maintenance activities employed in the crypto farms. We will see these activities from different perspectives. It includes the standard inspection procedures on the miner and the various troubleshooting techniques on the miner. It will help you to understand all the activities in a better way. We will also see the notices for going on and off racks.
We will use the example of a mining farm that uses Antminers for crypto mining. But you can also use this guide for all other mining farms that use other types of miners. So, let us get started.
The initial section of the crypto mining farm’s operation and maintenance is the routine inspection. There are multiple types of inspections involved here. It includes power inspection, an inspection of the network, and an inspection of hashrate. All these inspection activities need to be done properly to ensure the smooth functioning of the mining process.
1. Inspection of Hashrate
You must check the hashrate of the system every 2 hours. You can use batch management software (BTC Tools) to check the hashrate. You can also check by logging in to the website of your mining pool. You can easily organize the different miners in the network with the help of the BTC Tool. You can view the temperature and the real-time hashrate using this tool. We recommend using the Google Chrome browser for ease of using the miners. You can set it as your default browser. It will allow you to bypass the filling of password and username. You can easily reach the mining status page by double-clicking the scanned miner. If you have altered the password of the miner, then you will have to reset it. Go to the BTC tools software program and click on “Settings” to change your password. You can go to the BTC website to download the BTC Tools software.
You must do regular checks in the miner to know any errors or omissions, if any. The checking must also be done on the main pool and the two spare mining pools. We strongly recommend changing the default password to avoid any unauthorized access by hackers. Never download any overclocking software or any other unauthorized firmware. These malicious codes can do more harm than good. You can use the APMiner Tool to batch modify the passwords. This tool is also capable of managing and scanning crypto miners. You can download the APMiner Tool from the antpool website.
Sometimes, you may spot miners with abnormal hashrate. If you find any such miners, you need to fix the issue as soon as possible. If you are not able to resolve the issue by yourself, consider contacting the manufacturer. You can go for an off the shelf repair or send it to the manufacturer.
2. Checking the Power for the Miner
You must check the voltage levels of the miner PDU socket on a daily basis. You can use a multimeter to check and ensure that the voltage is within the acceptable range. Different countries will have different power grid specifications. Some use the 220V AC grid, whereas others have the 110V AC grid. So, you must cross-check accordingly. If there is an excess drop or increase in voltage, it will make the miner unstable. It can even lead to the failure of the miners. If you find any issues with voltage levels, call your electrician person for maintenance and rectify the problem.
You must also check the power distribution cabinet and ensure that the 3-phase current meter is working correctly. You must do it daily. Check if there is a deviation above 15 percent on either side. If any, tell your electrician to check the balance of the 3-phase load. The higher voltage on any of the phases is mainly due to the imbalance. You need to ensure that the 3-phase connection is properly distributed among the different miners. Ensure even distribution of the power supply.
There can be electric leakages in the miner. So, you need to check the leakage on a regular basis. Most of the electric leakage issue with the miners is due to the poor grounding scheme. Improper grounding can lead to a faulty hashboard and control board. Thus, you may end up investing more in the boards. So, always use the service of an electrician to check the grounding scheme. Make sure that there is a good grounding scheme that avoids electric leakage.
Electric Leakage Detection Methods
There are two methods that you can use to detect electric leakage.
The first method is used when the miner is still in operation. Here, you will have to measure the potential difference on the case of the miner equipment and the shelf. You can use the multimeter for the voltage checks. If you find any voltage, it shows poor grounding. You can use this method only when the miner case is not having any kind of contact with the metal shelf.
You need to measure the potential difference in a different section in the second method. Here, the potential difference needs to be checked on the case of the miner and the port’s metal cover. If the multimeter shows any voltage reading, it indicates a poor grounding scheme. You can use this method only when the control board is not touching the miner network port’s metal cover.
The voltage reading on the multimeter must be below 1V for these voltage checks. If it is above 1V, you need to do a proper grounding.
There should not be any high grounding resistance between the socket of the miner and the grounding wire connect to the shelf. This resistance value must be less than 4 Ohms. You may require the service of an electrician if you find any issues in grounding. You must do it as soon as possible. Sometimes, you may feel some numbness in the power supply used for mining. If there are any, you need to check the grounding cable. You must check the aging of the cables and wires used in the system regularly. The standard procedure also demands for checking the temperature of the cables, breaker, connectors, and flat copper cables. Get an infrared thermometer for contactless temperature checks on these parts. Measure the temperature on the flat cables and the copper nose. If it is more than 70 degrees Celsius, you will have to get the service of your electrician.
You need to avoid frequent power outages in the crypto mining farm. Stable power is required for the smooth functioning of a mining farm. When you are doing power outages, you must do it in a planned fashion. First, you need to start switching off the lowest air switch. The master switch needs to be turned off at last. It should be a step-by-step process. Once the power is back, you need to ensure that all the switches in the individual miners are turned off. You need to switch on the master switch first and then the individual miner power switch. Otherwise, the voltage surge in the system may damage the miners.
3. Checking the Miner Network
You need to check the network on the router or the computer connected to it regularly. Never use the same router for your daily life things and mining Use two separate routers for each of these activities. It is to avoid network risks from unknown websites. You can also expect a fluctuation in the hashrate when you use the mining network for daily life stuff. The fluctuation occurs when you stream a video or download any big file. If you have no separate router option, you might have to share the network. In such cases, you can limit the IP addresses used by the home life stuff. It helps to avoid unknown website access.
You need to do regular pings on the crypto mining polls utilized by the miners. This ping is to check the network delay, if any. The delay threshold limit is 100 ms. If the delay is above this value, you will have to check the external and internal networks and fix it. The network administrator can check the LAN to ensure the network device’s status. The ISP can check for the lines and equipment of the external network. There will be fluctuation in the hashrate if there is a considerable delay in the network. You can find the option to ping on the Network Diagnostics page. You can enter the mining pool URL in the box and click the ping button to see the time delay. If you try to ping without accessing the current mining pool, it shows network failure. You can use it to check the network when you are getting a zero hashrate.
You need to monitor the temperature of the network equipment on a daily basis. Use the infrared thermometer to measure the temperature. You may have to install a fan or change the current position of the network device if you see a temperature above 45 degrees Celsius. Never place the network device and the cables in the vicinity of the miner’s air outlets. It will increase the temperature and affect the data flow. The temperature of the CPU must not go beyond a certain value. If it gets too hot, there can be instability in the network. It can even result in a crash. You must ensure a high-quality main network. We strongly recommend to power off the miner completely when there is a power cut for more than fifteen minutes. The power consumed by the crypto miner is proportional to the hashrate. If you are getting a zero hashrate or an insufficient one, you can see a relative drop in the consumption of power.
4. Checking the Humidity and Temperature
You need to measure the temperature at the air inlet of the miner on a daily basis. Use the infrared thermometer for the temperature measurement. You need to ensure that the temperature is always between 10 and 25 degrees Celsius. If the temperature at the air inlet of the crypto miner crosses 30 degrees Celsius, the high-temperature protection circuitry will be triggered. Thus, the hashrate will become zero. So, if there is an increase in temperature, you need to lower the temperature of the environment. Sometimes, there can be a hot air reflow in the miner. So, you need to check that also. When you are using external fans in the farm for the miners, the fans need to be aligned in the same direction as the internal fans of the miner. Otherwise, it will result in a heat buildup. It occurs when they are in the opposite direction. So, position the external fans according to the orientation of the fans of the miner. The high-temperature protection feature of the miner ensures that the miner is not damaged at a higher temperature. The miner will resume its normal operation once the temperature comes down. Now, you can see that the hashrates are resumed to their actual values. When you are setting up a mining farm, you need to have some set up to control the room temperature. Some of the techniques that you can use to lower the room temperature include the use of negative pressure fans, water curtains, and thermal isolation. You can use baffles or other similar materials to prevent the backflow of air in the air inlet area of the miner.
You need to measure the humidity at the air inlet of the miner on a daily basis. Sometimes you can see water vapor getting condensed on different parts of the miner. So, you need to check for such condensations also. If the humidity meter shows a reading above 65 percent, you need to bring it down. If you are using the water curtain technique to lower the ambient temperature, it can result in an increase in humidity if the shelf is nearby. When you use the water curtain system, you can maintain a 2 to 3 meter distance to avoid this increase in humidity. But make sure that the curtains are not too far. Otherwise, it will affect the efficiency of cooling. Also, if the moisture exists for a long time and the quality of the air inside is poor, it will result in the corrosion of the hashboards. Thus, you will end up damaging the hashboard. It can happen if there are any big chemical plants near the mining farm. So, never operate a crypto mining farm in a coastal area. The alkaline particles in the air will oxidize and damage the boards of the miner.
Water leakage from the water curtains can be a severe issue. So, you need to check for any leakage to the ground. If there are any blocks created due to the alkaline residue, clear them immediately. Make sure that there is an even flow of water in the water curtains. Otherwise, there will be an increase in heat inside, and it will impact the hashrate of the mining process. You need to check the level of water and the pump on a regular basis.
If your mining farm is experiencing an extremely harsh winter, you need to turn on the heating fans to regulate the temperature in the mining farm.
5. Inspecting the Miner Environment
You will have to inspect the status of the negative pressure fans daily. It needs to be in a fully functional state. You must clean the mining farms regularly to ensure no paper or dust particles.
You need to inspect the miners regularly for dust on a monthly basis. You can remove the front fan of the miner for proper inspection. If you find lots of dust particles in the heat sink, then it is an indication of proper cleaning to be done. Otherwise, the accumulated dust will trigger temperature protection. Never use any hard brush to remove the dust from the heat sink. If you use a hard brush, the sink will fall off. Instead, gently brush the heat sink and blow off the dust particles. Make sure to use an antistatic brush for dust removal. Otherwise, ESD can cause issues to the electric components in the miner.
If there are lots of dust particles and heavy sands, you will have to do a weekly inspection of the dust mesh. Use a vacuum cleaner to do the weekly cleaning. If your fam is located in a place with lots of sand and dust, you need to take lots of measures to ensure that this dust doesn’t enter inside the mining farm.
If rain or water enters the crypto mining farm, it can damage the crypto miners. So, take proper measures to avoid the entry of water into your mining facility.
6. A Thorough Visual Inspection
You need to do a thorough checking on the air inlet of the miner for paper scraps or garbage. If you find any, remove them immediately. It is essential to ensure proper airflow into the miner. The accumulation of dust near the air inlet can lead to zero hashrate due to the high-temperature protection feature.
There will be indicators on the miner that shows its health status. You need to check these indicators on a daily basis. If any miner shows red light, you will have to take appropriate measures to fix the issue. You can inspect the two fans of the miner at first. If the fans are not working or are damaged, you will have to replace them with new ones. Go to the mining status page in the kernel to check the fan’s running status. When you install the new fans, ensure that they are installed in the correct orientation. If you do it in the opposite way, the hot air will not get pushed out of the crypto miner. It can damage the miner.
You need to inspect the health of the network cables on a regular basis. If any of the cables are damaged, you need to replace them. Also, check for any overheating in the cables. We don’t recommend using power splitters. It can result in an overload. If you are in the 220V AC grid, we recommend using a power cord with a minimum 10A rating, and the length of the cord must be less than 1.5m.
Going On and Going Off Rack Notices
We have notices for Going Off racks as well as notices for Going On Racks. We have already mentioned it in the first part.
1. Notices for Miners Going On Racks
You need to ensure that no violent collisions are happening on the surface of the crypto miner. If there are any collisions, then there is a chance that the heat sinks in the miner may fall. It will result in heating of the miners. So, check and ensure that the heat sinks are in their actual place. You can also find couple of fans in the miner. Make sure that they are also in place.
You need to ensure proper plugging of the flat cables, including the power supply cable, fan cable, and hashboard cable. You will have to do it before going on the racks. Makes sure that the 6-pin cable that connects the hashboard is inserted in the correct orientation. You must also ensure that the power cord is inserted properly into the control board of the miner. It is essential to ensure proper power delivery to the miner system. If not, it will result in damage to the hashboard. Make sure there are no loose connections. Loose connections can burn the miner. Poor sockets can result in loose connections. So always invest in high-quality power sockets for your grid.
When you are moving the crypto miners from one place, make sure to do it gently and in a safe manner. Never hold the control board or hash boar by hand. You may damage the electronic circuitry in the board when you hold it with your hand. If you are using the service of porters for the movement of miners, you need to tell them to handle the boards carefully. If the miners are not handled properly, the heat sinks inside may get dropped, and the fans can get damaged. The boards can also get damaged by mishandling.
You can use the BTCTools software program or the APMiner Tool to alter the mining workers when the crypto miners are placed in the racks. You can use the static IPs as and when needed. You must also check and ensure that the hashrate that you are getting is normal or not. You can use the original shelf position as the IP address when you place single crypto minors on the racks.
When you are placing many crypto miners in batches, you need to have proper planning about the IP addresses. You can assign subsequent static IP addresses. You must separate the static IP from the DHCP. It is better to explain it with an example. Consider this example, suppose there are 150 crypto miners in the racks. You can set the gateway to 22.214.171.124. You can select the DHCP as 126.96.36.199-240. So, you can use all the reserved IPs for the coming static IP address. In most cases, one shelf will be managed by one network. You can employ the services of APMiner Tool to assign static IP addresses for the miners. You can click on the “Configure IP” button to set it.
2. Notices for Miners Going Off Racks
If you see any faults in the miner, remove it from the rack to see if the fault is still there or not. There are many issues that can be solved onsite. It includes the issues like fan failure, high-temperature protection, system failure, and network failure. It is always good to stock up some spare fans for the miners on site. Thus, you can replace the fans easily whenever it goes bad. Always ensure that there is some severe fault of the miner which cannot be fixed by you before sending it back to the manufacturer. Such misjudgments can result in loss of time and money. If the failure is related to the fans, you can check the kernel logs. You can view the error here.
Always make sure that the IP address you are using corresponds to the crypto miner. It is a common mistake seen in many mining farms. If you are not able to confirm the IP of the miner, you can make use of the IP reporter software program. Click on the Start button and press the Miner OP button to confirm the IP address. Otherwise, you can use the APMiner tool and click the” Setting IP” button. Click on the IP button on the minor to check and confirm.
When you move the crypto miners on and off the racks, you need to ensure that the miners are in the powered-off state. Handle it carefully. Never try to carry the hashboard by holding on to its power cord or cable alone. Never throw the miner boards or cables from a height.
Some mining farms will have the provision of thermal isolation on the shelves. If you have such isolations, then when you remove the miner, there will be empty holes. Such holes must be blocked. It is for avoiding the hot air reflux. Once you remove the miner from the racks, you need to remove the dust and moisture present in it. After treatment, the miner must be placed in a moisture-free and dust-free area before shipping it back for maintenance.
Troubleshooting the Crypto Miner
There are different types of miner faults. These faults arise due to multiple reasons. Whenever you see a fault, start with a simple solution. The simple solutions include upgrading the firmware, inspecting the network, inspecting the cable, replacement of power supply, lowering the ambient temperature, and removing the dust particles. If you are still not able to solve the issue after doing all these activities, you need to look for the power checks and the environment checks. If the problem is still there, you need to contact the manufacturer and ship the miner back to them for maintenance.
For troubleshooting and solutions for Antminer, please refer to related articles of EastShore:
- Troubleshooting for Antminer: hashing board | Inovatec Mining Devices
- Troubleshooting for Antminer: controller & fan | Inovatec Mining Devices
- Troubleshooting for Antminer: Common External Problems | Inovatec Mining Devices
- Miner Virus Prevention and Solutions | Inovatec Mining Devices
For faults and solutions of other types of miners, please consult with the manufacturers.