Efficient Bulk File Ingestion into Data Lake using SMB and FTP protocols

Abstract

It is very essential for companies in the retail industry to exchange sensitive information like Purchase Orders, Invoices, Payments Information, Contracts and other types of information for improving efficiency, reducing costs, and for strengthening business relationships. The sensitive information also gets exchanged between different teams within an organization for better collaboration, efficiency and to extract valuable insights. This paper introduces a cost effective and secure approach which is tailored for bulk ingestion into databricks data lake using SFTP and SMB protocols. The proposed file transfer framework makes use of python packages like paramiko (which implements the Secure Shell SSHv2 protocol for secure file transfer over unsecured network) and pysmb(which implements SMB protocol for secure file sharing across different operating systems within a network) to transfer the files into the databricks volume for further processing by transformation pipelines in databricks cluster. Using this methodology various types of file transfers like synchronous small file transfer, asynchronous large file transfer and continuous file transfer can be achieved. Deletion of files on the remote server after a configured number of successful transfers is also achieved using the proposed methodology. The key findings of this implementation are as following: simplified setup procedure, less development effort and decent performance. The findings suggest that the proposed framework with data integrity and remote file management capabilities is a secure, simplified, versatile and a reliable choice for transferring sensitive files.