- Hands-On Artificial Intelligence for IoT
- Amita Kapoor
- 154字
- 2021-07-02 14:02:02
Using hdfs3 with HDFS
hdfs3 is a lightweight Python wrapper around the C/C++ libhdfs3 library. It allows us to use HDFS natively from Python. To start, we first need to connect with the HDFS NameNode; this is done using the HDFileSystem class:
from hdfs3 import HDFileSystem
hdfs = HDFileSystem(host = 'localhost', port=8020)
This automatically establishes a connection with the NameNode. Now, we can access a directory listing using the following:
print(hdfs.ls('/tmp'))
This will list all the files and directories in the tmp folder. You can use functions such as mkdir to make a directory and cp to copy a file from one location to another. To write into a file, we open it first using the open method and use write:
with hdfs.open('/tmp/file1.txt','wb') as f:
f.write(b'You are Awesome!')
Data can be read from the file:
with hdfs.open('/tmp/file1.txt') as f:
print(f.read())
You can learn more about hdfs3 from its documentation: https://media.readthedocs.org/pdf/hdfs3/latest/hdfs3.pdf.
推薦閱讀
- 傳感器技術(shù)實驗教程
- Hands-On Data Science with SQL Server 2017
- 返璞歸真:UNIX技術(shù)內(nèi)幕
- Mastering Salesforce CRM Administration
- 數(shù)據(jù)運營之路:掘金數(shù)據(jù)化時代
- 工業(yè)機器人安裝與調(diào)試
- Machine Learning with Apache Spark Quick Start Guide
- 智能生產(chǎn)線的重構(gòu)方法
- Excel 2007常見技法與行業(yè)應(yīng)用實例精講
- R Data Analysis Projects
- Visual Basic項目開發(fā)案例精粹
- ADuC系列ARM器件應(yīng)用技術(shù)
- 納米集成電路制造工藝(第2版)
- Practical AWS Networking
- 工業(yè)機器人編程指令詳解