官术网_书友最值得收藏!

Using hdfs3 with HDFS

hdfs3 is a lightweight Python wrapper around the C/C++ libhdfs3 library. It allows us to use HDFS natively from Python. To start, we first need to connect with the HDFS NameNode; this is done using the HDFileSystem class:

from hdfs3 import HDFileSystem
hdfs = HDFileSystem(host = 'localhost', port=8020)

This automatically establishes a connection with the NameNode. Now, we can access a directory listing using the following:

print(hdfs.ls('/tmp')) 

This will list all the files and directories in the tmp folder. You can use functions such as mkdir to make a directory and cp to copy a file from one location to another. To write into a file, we open it first using the open method and use write:

with hdfs.open('/tmp/file1.txt','wb') as f:
f.write(b'You are Awesome!')

Data can be read from the file:

with hdfs.open('/tmp/file1.txt') as f:
print(f.read())

You can learn more about hdfs3 from its documentation: https://media.readthedocs.org/pdf/hdfs3/latest/hdfs3.pdf

主站蜘蛛池模板: 威远县| 虹口区| 拜泉县| 庆阳市| 遂川县| 怀仁县| 台南市| 清镇市| 集贤县| 淮阳县| 闸北区| 会泽县| 崇阳县| 静安区| 得荣县| 莫力| 缙云县| 兴文县| 博客| 宜阳县| 泸西县| 罗江县| 北辰区| 文水县| 云梦县| 凤冈县| 康马县| 铁岭县| 海伦市| 丹阳市| 肃宁县| 澳门| 印江| 旺苍县| 山阴县| 屏东市| 西畴县| 长兴县| 越西县| 泸州市| 上栗县|