官术网_书友最值得收藏!

Using hdfs3 with HDFS

hdfs3 is a lightweight Python wrapper around the C/C++ libhdfs3 library. It allows us to use HDFS natively from Python. To start, we first need to connect with the HDFS NameNode; this is done using the HDFileSystem class:

from hdfs3 import HDFileSystem
hdfs = HDFileSystem(host = 'localhost', port=8020)

This automatically establishes a connection with the NameNode. Now, we can access a directory listing using the following:

print(hdfs.ls('/tmp')) 

This will list all the files and directories in the tmp folder. You can use functions such as mkdir to make a directory and cp to copy a file from one location to another. To write into a file, we open it first using the open method and use write:

with hdfs.open('/tmp/file1.txt','wb') as f:
f.write(b'You are Awesome!')

Data can be read from the file:

with hdfs.open('/tmp/file1.txt') as f:
print(f.read())

You can learn more about hdfs3 from its documentation: https://media.readthedocs.org/pdf/hdfs3/latest/hdfs3.pdf

主站蜘蛛池模板: 涟源市| 色达县| 南通市| 博白县| 霍城县| 滦平县| 彩票| 昌平区| 宝坻区| 莎车县| 布拖县| 土默特左旗| 长丰县| 绩溪县| 寿阳县| 武鸣县| 东阿县| 阜平县| 潜山县| 嵩明县| 松滋市| 克什克腾旗| 合阳县| 武夷山市| 嘉定区| 瑞丽市| 洪江市| 耿马| 咸丰县| 枣强县| 信丰县| 平塘县| 准格尔旗| 镶黄旗| 龙海市| 汨罗市| 红河县| 定西市| 桃园县| 论坛| 滨海县|