Python實(shí)現(xiàn)遍歷讀取文件或文件夾
搞機(jī)器學(xué)習(xí)或者深度學(xué)習(xí)算法很多時(shí)候需要遍歷某個(gè)目錄讀取文件,特別是經(jīng)常需要讀取某個(gè)特定后綴的文件,比如圖片的話可能需要讀取jpg, png, bmp格式的文件。python本身的庫(kù)函數(shù)功能沒(méi)有這么定制化,所以就需要再重新包裝一下。
例子
假設(shè)我們有如下的目錄結(jié)構(gòu),以bmp結(jié)尾的是文件,其他是文件夾。下面的程序都將以該目錄結(jié)構(gòu)為例進(jìn)行說(shuō)明。
os.listdir
os.listdir僅讀取當(dāng)前路徑下的文件和文件夾,返回一個(gè)列表。讀取demo目錄結(jié)構(gòu)的代碼和結(jié)果如下:
path = r'D:\data' items = os.listdir(path) # ==> ['1.bmp', '2.bmp', 'a', 'b']
os.walk
os.walk本身已經(jīng)是遍歷讀取,包含所有的子文件(夾)但是其結(jié)果不像是os.listdir一樣是個(gè)list,而是一個(gè)比較復(fù)雜的數(shù)據(jù)體,難以直接使用,所以一般需要再處理一下。我們可以使用for語(yǔ)句將其打印出來(lái)看看:
path = r'D:\data' # part 1 for items in os.walk(path): print(items) # part 2 for main_dir, sub_dir_list, sub_file_list in os.walk(path): print(main_dir, sub_dir_list, sub_file_list)
結(jié)果為:
# part 1
('D:\\data', ['a', 'b'], ['1.bmp', '2.bmp'])
('D:\\data\\a', [], ['a1.bmp'])
('D:\\data\\b', [], ['b1.bmp'])
# part 2
D:\data ['a', 'b'] ['1.bmp', '2.bmp']
D:\data\a [] ['a1.bmp']
D:\data\b [] ['b1.bmp']
使用迭代器對(duì)os.walk()的結(jié)果進(jìn)行輸出,發(fā)現(xiàn)每一條包含三個(gè)部分(part 1),在part 2中,我們給三個(gè)部分分別起名為main_dir, sub_dir_list, sub_file_list,下面對(duì)其進(jìn)行簡(jiǎn)單解釋:
- main_dir:遍歷得到的路徑下所有文件夾
- sub_dir_list:main_dir下面的文件夾
- sub_file_list:main_dir下面的文件
連接main_dir和sub_file_list中的文件可以得到路徑下的所有文件。
sub_dir_list在這里則沒(méi)有用處,我們無(wú)需再去遍歷sub_dir_list,因?yàn)樗鼈円呀?jīng)包含在main_dir里了。
遍歷讀取代碼
代碼邏輯如下:
需要有后綴辨別功能,并且能夠同時(shí)辨別多個(gè)后綴
需要有遞歸和非遞歸功能
返回的是以入?yún)ath為前綴的路徑,所以如果path是完整路徑那么返回的就是完整路徑,否則就不是
# -*- coding: utf-8 -*- import os def file_ext(filename, level=1): """ return extension of filename Parameters: ----------- filename: str name of file, path can be included level: int level of extension. for example, if filename is 'sky.png.bak', the 1st level extension is 'bak', and the 2nd level extension is 'png' Returns: -------- extension of filename """ return filename.split('.')[-level] def _contain_file(path, extensions): """ check whether path contains any file whose extension is in extensions list Parameters: ----------- path: str path to be checked extensions: str or list/tuple of str extension or extensions list Returns: -------- return True if contains, else return False """ assert os.path.exists(path), 'path must exist' assert os.path.isdir(path), 'path must be dir' if isinstance(extensions, str): extensions = [extensions] for file in os.listdir(path): if os.path.isfile(os.path.join(path, file)): if (extensions is None) or (file_ext(file) in extensions): return True return False def _process_extensions(extensions=None): """ preprocess and check extensions, if extensions is str, convert it to list. Parameters: ----------- extensions: str or list/tuple of str file extensions Returns: -------- extensions: list/tuple of str file extensions """ if extensions is not None: if isinstance(extensions, str): extensions = [extensions] assert isinstance(extensions, (list, tuple)), \ 'extensions must be str or list/tuple of str' for ext in extensions: assert isinstance(ext, str), 'extension must be str' return extensions def get_files(path, extensions=None, is_recursive=True): """ read files in path. if extensions is None, read all files, if extensions are specified, only read the files who have one of the extensions. if is_recursive is True, recursively read all files, if is_recursive is False, only read files in current path. Parameters: ----------- path: str path to be read extensions: str or list/tuple of str file extensions is_recursive: bool whether read files recursively. read recursively is True, while just read files in current path if False Returns: -------- files: the obtained files in path """ extensions = _process_extensions(extensions) files = [] # get files in current path if not is_recursive: for name in os.listdir(path): fullname = os.path.join(path, name) if os.path.isfile(fullname): if (extensions is None) or (file_ext(fullname) in extensions): files.append(fullname) return files # get files recursively for main_dir, _, sub_file_list in os.walk(path): for filename in sub_file_list: fullname = os.path.join(main_dir, filename) if (extensions is None) or (file_ext(fullname) in extensions): files.append(fullname) return files def get_folders(path, extensions=None, is_recursive=True): """ read folders in path. if extensions is None, read all folders, if extensions are specified, only read the folders who contain any files that have one of the extensions. if is_recursive is True, recursively read all folders, if is_recursive is False, only read folders in current path. Parameters: ----------- path: str path to be read extensions: str or list/tuple of str file extensions is_recursive: bool whether read folders recursively. read recursively is True, while just read folders in current path if False Returns: -------- folders: the obtained folders in path """ extensions = _process_extensions(extensions) folders = [] # get folders in current path if not is_recursive: for name in os.listdir(path): fullname = os.path.join(path, name) if os.path.isdir(fullname): if (extensions is None) or \ (_contain_file(fullname, extensions)): folders.append(fullname) return folders # get folders recursively for main_dir, _, _ in os.walk(path): if (extensions is None) or (_contain_file(main_dir, extensions)): folders.append(main_dir) return folders if __name__ == '__main__': path = r'.\data' files = get_files(path) print(files) # ==> ['D:\\data\\1.bmp', 'D:\\data\\2.bmp', 'D:\\data\\a\\a1.bmp', 'D:\\data\\b\\b1.bmp'] folders = get_folders(path) print(folders) # ==> ['D:\\data', 'D:\\data\\a', 'D:\\data\\b']
以上就是Python實(shí)現(xiàn)遍歷讀取文件或文件夾的詳細(xì)內(nèi)容,更多關(guān)于Python遍歷讀取文件的資料請(qǐng)關(guān)注腳本之家其它相關(guān)文章!
相關(guān)文章
Python將多個(gè)excel表格合并為一個(gè)表格
這篇文章主要為大家詳細(xì)介紹了Python將多個(gè)excel表格合并為一個(gè)表格的方法,文中示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2018-01-01python3實(shí)現(xiàn)小球轉(zhuǎn)動(dòng)抽獎(jiǎng)小游戲
這篇文章主要為大家詳細(xì)介紹了python3實(shí)現(xiàn)小球轉(zhuǎn)動(dòng)抽獎(jiǎng)小游戲,文中示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2019-05-05python網(wǎng)絡(luò)爬蟲selenium打開多窗口與切換頁(yè)面的實(shí)現(xiàn)
本文主要介紹了python網(wǎng)絡(luò)爬蟲selenium打開多窗口與切換頁(yè)面的實(shí)現(xiàn),文中通過(guò)示例代碼介紹的非常詳細(xì),具有一定的參考價(jià)值,感興趣的小伙伴們可以參考一下2022-01-01scrapy爬蟲遇到j(luò)s動(dòng)態(tài)渲染問(wèn)題
本文主要介紹了scrapy爬蟲遇到j(luò)s動(dòng)態(tài)渲染問(wèn)題,通過(guò)js渲染出來(lái)的動(dòng)態(tài)網(wǎng)頁(yè)的內(nèi)容與網(wǎng)頁(yè)文件內(nèi)容是不一樣的,本文就來(lái)解決一下這些問(wèn)題,感興趣的可以了解一下2022-05-05Python辦公自動(dòng)化之?dāng)?shù)據(jù)預(yù)處理和數(shù)據(jù)校驗(yàn)詳解
這篇文章主要為大家詳細(xì)介紹了Python辦公自動(dòng)化中數(shù)據(jù)預(yù)處理和數(shù)據(jù)校驗(yàn)的相關(guān)知識(shí),文中的示例代碼講解詳細(xì),感興趣的小伙伴可以參考一下2024-01-01